Review Comment:
In this paper, the authors introduce two algorithms for automatically providing summaries of ontologies. They have tested their approach against an existing set of ontologies already used in several past works (e.g., [9] and [10]), which are included in the evaluation. Finally, they also provided a section where they present a possible approach to select relevant concepts in the original ontology that may be relevant for exploring it starting from specific nodes included in the related summary.
This is a work that have been chosen by the organizing committee of the Extended Semantic Web Conference 2015 to be published in the corresponding special issue organized at the Semantic Web Journal. Thus, the authors had to provide an extension that should include new valid research contributions and insights of the overal work. However, by reading the paper, the actual new original contribution is not enough to deserve a publication in the journal.
In particular, they added:
- Definition 8;
- The computational cost of all the algorithms proposed;
- Section 4.2, where they introduce the new algorithm proposed;
- The results of the new algorithm, which have been added to the evaluation;
- A new metric for calculating the relevance of a summary and a related comparison;
- A sketched approach for exploring the full ontology starting from the concept in the summary.
What it is needed for having an appropriate extension to deserve a publication in the ESWC 2015 special issue is:
- Extending in an appropriate way the evaluation
* by including at least a variant of the first algorithm presented by the authors (which should include blank nodes);
* by using the algorithm presented in [10] in all the evaluations (including CIDOC-CRM with and without instances, and the execution times);
* by preparing a new evaluation - with a new user study - for analysing also the properties their algorithms return in the summary, which is not analysed and evaluated at all in the paper;
* by evaluating the metrics presented in section 8.
- Correct several conceptual errors in the paper about the nomenclature they used to describe ontologies (i.e., RDFS vs. OWL) and blank nodes, unsupported claims, and the features of the other algorithms involved in the experiment, w.r.t. [10] in particular;
Please find detailed comments as follows.
- There are some inconsistencies and issues in the content of the paper. First of all, the authors say that their focus is on RDFS ontologies, and explicitly highlight the fact that the handling of OWL ontologies is left to future works. However, what the authors actually are saying is that they consider only RDFS-like concepts/relations of the analysed ontologies (i.e., class, properties, sub-class hierarchies, sub-property hierarchies, domain and range classes), without caring about any additional construct that Semantic Web ontologies can made available, such as class restrictions, property characteristics, etc. This doesn't mean they consider only RDFS ontologies – in fact several of the ontologies they have used in their experiments are, indeed, OWL ontologies –, but simply that they just consider a limited set of axioms in their analysis. This should be clarified.
- While developing an approach for such limited set of axioms is totally fine for producing an ontology summary, it is not true at all that RDFS is the de-facto standard for publishing and representing data on the web. First of all, I've not found any strong evidence of this claim in the cited work (i.e., [3]). Second, a good part of the vocabularies included in [3] are actually OWL ontologies.
- Again, about the use of blank nodes, there are some aspects that should be clarified as well. OWL ontologies usually adopts blank nodes for expressing class restrictions, group disjointness, etc. However, since these aspect are not considered at all in the algorithms proposed by the authors – at least, as far as I understood – it is not clear what they refer to with "black nodes". Can they refer to individuals of a certain class that are not provided with a URL? This aspect should be explicitly discussed in the paper.
- The sentence (after definition 1) "if p is a property, the triple {p, type, property} can be inferred" seems a bit tautological to me, since to say (in RDFS) that p is a property one has to write exactly that triple – it is not an inference at all. Did the authors want to say that if p is used as a predicate in a triple, then p can be inferred as a property?
- Related to definition 1 and 2, it is not clear where the property rdf:type is involved in. To me, it should be used in both the schema graph and the instance graph. However, it seems it is used only in the latter one. This part should be clarified.
- In section 3.2.1 the author say that the weights of the in-centrality formula have been experimentally defined. How?
- I would suggest to split definition 2 in two (let's say def 2a and def 2b) so as to clearly define in and out centrality formulas. It is quite difficult to follow such text in the present form.
- In definition 8, the situation described in point 3 is always true to me: there exists always a path from a concept to another by going through the top concept (e.g., owl:Thing for classes) by means of the sub-class relations.
- Why did not the authors implement the blank node handling also for the first algorithm? How better would it work if blank nodes are then taken into account? I would like to see in the evaluation also the use of this kind of improved version of first algorithm – it should not be too much difficult to be implemented.
- In section 4.2 the authors refer to Definition 6, while I think it should be definition 9.
- I strongly suggest to provide examples for explaining the execution of the two algorithm graphically, it could be very useful for improving the understandability of the process.
- I've tried to run the tool provided in the URL in footnote 2, but it doesn't work with both the ontologies used in the evaluation nor any other ontology available online. This is an issue for the reproducibility of the experiment.
- There are some misinterpretation of some of the algorithms used in the comparison, in particular with the Key Concept Extractor (KCE) algorithm [10]. First, the fact that the ontologies Biosphere, Financial and AKTOR have been used to evaluate past works on RDFS summarisation is just wrong, since they are OWL ontologies indeed – I think that here the problem is about the "nomenclature" used by the authors, see my comment above about it. Second, the claim that the authors cannot uses ontologies with instances since the other methods do not consider them is simply false for KCE. In fact, the density of such algorithm does use individuals of concepts in its formula – which are specified with a quite low weight in [10] (even if the algorithm is parametric), but still they are used. The fact that the ontologies used in [10] for the evaluation don't include any individual doesn't mean that the algorithm is not able to process them. Thus, I strongly suggest to the authors to include also KCE in the evaluation with individuals. Note that the algorithm is actually implemented and released with an open source license (https://github.com/essepuntato/kce). In addition, it is also available as a web application (http://www.essepuntato.it/kce).
- After the similarity formula, it is not clear what "the set of classes" actually refers to. Are those classes in A or in S?
- The fact that, in the similarity formula, the authors give more importance to superclasses instead of subclasses is not supported by any strong analysis. What would it happen if we consider sub and super classes in the same way (let's say 0.5 and 0.5)? What would it happen if we give more consideration to sub classes instead of super classes? All these variants seem quite reasonable to me. I suggest to the authors to consider all of these different similarity measures in their analysis, or to provide a strong explanation of why superclasses deserve more importance.
- The fact that the authors use the CIDOC-CRM core as summary of the full CIDOC-CRM is not convincing at all, and rather seems a simplification to me. Since the selection of the summaries for the other ontologies has been done by humans following a particular protocol, I would like to see a similar approach also for the CIDOC-CRM one, so as to basically compare summaries that have been done in the same way. In addition, that will result in a clear additional contribution of the work. Note that, since the source of KCE is available, the authors should also consider at least that method in the evaluation of CIDOC-CRM.
- There is no evidence of "low quality" in [18] related to the use of black nodes in ontologies. Actually, blank nodes are used to define class restriction, and thus are rather useful and surely not so low quality.
- While a great work has been done by the authors from a pure algorithmic perspective about including properties in the summary they produce, there is no evaluation about them in the paper. This basically means that the summary returned by authors' algorithms are evaluated only partially. I totally understand that providing an evaluation like that means additional work - they should have to involve users, asking them to provide summaries which also include also properties. However, I firmly believe that this contribution is necessary for deserving acceptance in the SWJ, since it would be a plausible extension of the conference paper for what concern the evaluation part.
- The execution time comparison (fig 8) should include also KCE, since the source is actually available online.
- In section 7, the authors claim that KCE [10] consider only hierarchical relationship. However, also properties associated to concept (i.e., those that have such concepts as domain) are considered in the computation - see the notion of density in the paper. In addition, still, KCE doesn't consider each node in isolation. In fact, there are a lot of metrics that are actually "local", i.e., that are computed taking into consideration the closest neighbours' value.
- The work in progress described in section 8 doesn't cite important references. In fact, studies for exploring an ontology starting from the summary provided by KCE have been done by running a quite huge user testing session in
Enrico Motta, Paul Mulholland, Silvio Peroni, Mathieu d'Aquin, José Manuél Gómez-Pérez, Victor Mendez, Fouad Zablith: A Novel Approach to Visualizing and Navigating Ontologies. International Semantic Web Conference (1) 2011: 470-486
by means of a tool, i.e., KC-Viz, that uses the output of the KCE algorithm for enabling novel ways for browsing and navigate ontologies:
Enrico Motta, Silvio Peroni, José Manuél Gómez-Pérez, Mathieu d'Aquin, Ning Li: Visualizing and Navigating Ontologies with KC-Viz. Ontology Engineering in a Networked World 2012: 343-362
In addition, the authors seem not to consider several important works done with the notion of dependency within the ontology domain, such as those presented by Del Vescovo et al., e.g.:
Chiara Del Vescovo, Damian Gessler, Pavel Klinov, Bijan Parsia, Ulrike Sattler, Thomas Schneider, Andrew Winget: Decomposition and Modular Structure of BioPortal Ontologies. International Semantic Web Conference (1) 2011: 130-145
In addition, the work the authors present in this section should be accompanied by an evaluation for proving its validity.
- In the last section of the paper, what does the authors mean with the fact that they will consider OWL in future developments of their algorithm? Will they consider class restrictions, property characteristics, disjointness, etc.? It is not clear in the paper.
|
Comments
Review comments
This paper describes two methods (1. SummaryCM, is a existing work; 2. SummaryRM, which is a new approach, with added support for blank nodes) for summarizing RDF/S knowledge bases. The authors claim that their methods gives better correlation to those summaries which were prepared by human experts. In my opinion, the work is good w.r.t. the relevance of the topic and its presentation. The approaches are described using formal notations are clearly explained. However, the manuscript requires a minor revision before publication.
One of my main concern is how far the first method is different from your previous work[11] -- though you have included algorithm complexity details etc--- I think a clear distinction or a summarization (of the details of the method) is necessary. Many of the contributions mentioned (at the end) of the Introduction (as the current work's contributions) coincide with your previous work[11].
"Specifically the contributions of this paper are the following:" // you may change this sentence."
"Our previous work could not handle blank nodes. However...." // you have given a positive appeal for including the blank nodes, but it turns out to have negative impact on the generated summaries. I think you should briefly cover this point in the introduction.
In addition, you should address the following issues:
*encountered many typos
*need rewriting of a few sentances, to give more clarity!
*a few notation and reference issues
*clarity of the algorithms used.
Detailed review:
---
Section 2: Preliminaries (Definition 1)
\lambda_{c} or \lambda_{C}??
\lambda_{p} or \lambda_{P}??
Please confirm.
---
Definition 2 (last paragraph): Font of the "C" (in c \in C) looks different.
---
Section 3.1: Reference 13 is misleading.
---
Section: 3.1.2: last paragraph
In "We conisder...the latter"
"the former" -- is missing
You meant the other way around?
Which is more important, user defined properties right!
Kindly give more clarity to the sentance: “ This is partly because the user defined properties correlate classes, each exposing the connectivity of the entire schema, in contrast to
the hierarchical RDF/S properties.”
---
Two sections with same titles (3.1 and 3.1.3) may confuse the reader.
---
In Algorithm-1 some functions look too abstract. For e.g., the fn. path_with_max_cov(B, S, vi) – provide more details.
---
“The correctness of the algorithm is proved by construction.” -- please give clarification.
---
Page-7 last paragraph.
To identify... the complexity "of" --missing-- its various components
---
Definition 8 and 10, “p” and “p'” should be italiticed uniformily.
---
“ Kruskal's greedy algorithm [16] is amongthe most efficient ones and we are using it in our
implementation.” --- efficent in what sense?
---
In Algorithm-2
What is “N” in line 7.
What is “r” in line 4.
“the result of our algorithm for a specific input is unique as well.” --- how is it possible? Are the relevance of all the propertes are unique?
---
Section-6
Intext repetitions of the class and property counts can be removied, since they are given in Table-1.
---
In Table-1, giving Property and User property counts togeather looks redudnant, sicne oneis the subset of the other. You may give RDF standard peoprty and User property counts (makes sense).
---
Section 6.2
reference [0] is misleading!
---
Eq. for Sim(.)
You may use A and R, for automatically produced summary and R for reference summary -- inproves readability.
---
Page-17 1st column last paragraph
“As we will show latter --later-- the way that this value increases as
the size of the summary becomes bigger gives us”
---
In Section 6.3: To evaluate.."these" four ontologies.
which?? the ontology names are too far.
Section 6.3
Paragraph-2:
“We have to note that whereas the reference summaries on these” -- rewrite
Section 6.3
Paragraph-2 ending: It would be very interesting if you could include some egs. of the selected classes / summary egs. as an Appendix section.
---
In Fig. 7 you have give statistics of the Bank-Ontology, you mean the Financial Ontology?? --- this mistake has happend at many other sections.
"bioshere" -- be consistent use "BIOSPHERE" or "Biosphere"
---
Section 6.3:
For example, the Aktors Portal ontology contains a huge amount of blank nodes, and when considered by the SummaryRM, as shown in Fig. 6, the quality of the result is worse than the summary created by SummaryCM. //rewrite -- need more clarity
---
Section 6.4:
use "bigger"... instead of “biggest”
---
Section 6.5:
Using comma (instead of period) is little confusing. 1,29 is 1.29 secounds or 129 secounds??
“Finally we can see that SummaryRM is more efficient that --- “than” ---SummaryCM since the latter has to assess for each” // efficency in term of what?
---
Section 7:
“these works (e.g. [9], [10]) provide a list of the more important nodes, whereas others [8], [9], [17] and our approach, create a valid summary schema.” // reference [9] is included in both the types??
---
Section 8:
1. italicize e in Def.9 “values of the *e*..”
2. The paragragh "In our running example,..... "E57 Material". is repetated in Page-15
3. In the paragraph above Definition-10:...according "to" their instances... -- "to" is missing
Some relevant related works are missing:
[1] Wu, G.; Li, J.; Feng, L.; and Wang, K. 2008. Identifying potentially important concepts and relations in an ontology. In International Semantic Web Conference, 33–49