A Quality Assurance Workflow for Ontologies based on Semantic Regularities

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
3 fair
== 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
1 very poor

Review
The paper proposes an extension to the RIO that now also includes guided detection of semantic regularities in addition to the syntactic ones, and is put in an overall 'quality assurance workflow' that also contains the Knowledge Explorer. This has been evaluated with a selection of SNOMED CT.

General comments
Overall, the idea proposed--detecting ontology design patterns [type: content patterns] from the ontology, not only based on explicit declared information, but also the derived knowledge--is interesting and worth pursuing for ontology authoring, both with respect to good design and finding modeling 'flaws/irregularities' (or, at least, sections that deviate from the content pattern). Unfortunately, it is not worked out well, certainly not presented well, and more useful evaluations and presentation thereof could have been done.
The detection of lexical patterns is from ref 11, the syntactic-RIO from ref 9, and I don't see why one needs the Knowledge Explorer over a standard reasoner (why not use a standard reasoner, save the deductions with the ontology, and feed that into the RIO?), so that then leaves the workflow picture and evaluation as described novelty.
'quality assurance workflow' is a bit of a big term when it includes only one method.
Regarding the evaluation, some tests have been done, but not the ones I would have expected. Now having not only syntactic but also semantic patterns is heralded as a useful addition, and the introduction promises an interesting evaluation ("We show that the analysis of the semantic patterns … not syntactic ones."). fine, then show me what extra does it give me. Or: why not evaluate by comparing patterns-based-on-syntactic only vs. syntactic+semantic vs. semantic? which ones did it find with the new RIO-approach that the syntactic-only-RIO did not? how many more did it find? Instead, what is shown, is some undiscussed data about clusters and whatnot and only one tiny paragraph buried at the bottom of p10 that has some relevant information that it finds substantially more instances of a selected pattern (more comments about evaluation below).

Detailed comments

The abstract is chaotic, even when re-re-reading it after having read the paper. It has syntactic regularities emphasized, but the targeted novelty is semantic regularities. then the irregularities pop up. regularities and patterns are used interchangeably (and doubly), likewise implicit information and entailments. Overall, it is a hodgepodge of terms without connecting sentences, and I'd suggest to stick to one set of terms without synonyms and stay with that at least in the abstract.

Introduction
While the authors may want to brand their work as regularities instead of ODPs, ODPs they are, and it would be fair to the ODP people to actually acknowledge their contribution (besides ref 1). Not granting SNOMED CT and FMA their reference isn't nice either.
Figure 1 introduces already a problem what I have with the other figures as well. It has a content design pattern under the term "regularity", which I get, but then in the "Example instantiation", it doesn't actually fully instantiate the regularity: aside from the typo of the position of the \exists, it just repeats stuff verbatim from the pattern, like the "(RoleGroup9(`Clinical course (attribute)' ", which then doesn't look like an instantiation at all.
having a section 1.1 on related work is odd there, and it misses a "in the remainder of this paper…". Also, it does not really strengthen your case: "The methods presented in this paper are unsupervised and are not limited to identifying the patterns existing in a catalogue.", but that was already the case with the RIO. Moreover, it is not about finding a pattern unsupervised--as there are 6 from the documentation you use, in itself also from a catalogue--but more about finding instances of a pattern.

Section 2
This section is a bit ambiguous on whether it was supposed to present the new workflow for quality assurance, as advertised in the list of contributions on pp2-3, or whether it is just your procedure of doing things, the latter reading of the section due to the "lastly, …" paragraph at the end of the section.

Section 4
The KE is fine, but, as mentioned above, it is not explained what crucial function it gives us that is really needed for finding the semantic (ir)regularities, as it just "allows client code to explore the completion tree built by a tableaux reasoner". I then thought perhaps it serves for inspecting a section, but the workflow description does not indicate that ("For the entailment generation we use KE with the whole ontology …")

Section 6.
In addition to aforementioned comments on the evaluation, the introduction paragraph is very high-level on what the methodology is for the evaluation carried out. Section 6.1 then continues with the six patterns identified elsewhere, and only after that the procedure for the evaluation is described. It would improve presentation and readability to re-order this.
The results and discussion of the "Comparison of syntactic and semantic regularities" is vague, at best. One can indeed see that there is a difference, by looking at the numbers in Table 1. There is no way, however, to ascertain the validity of the authors' statement that "The semantic regularities are more uniform" and that "the entities with the lexical patterns are distributed in fewer clusters and the expected patterns are more well formed in most cases", nor that there are "fewer variations on the generalisations that refer to an expected pattern.". The table doesn't say much at all, or at least is not as self-explanatory as the authors may have deemed it.
While no data is given about the irregularities, findings such as "14% of the chronic terms do not instantiate the documented pattern" are interesting and potentially useful for improving the quality of an ontology, and "Only 2% … while a remaining 84%…" even more (though that's about regularities, not irregularities). It probably comes from the data in table 1, but when I take the lexical pattern for chronic as the max (1219) and 1053 as the semantic regularities instantiations of the documented pattern found, then with 1053/1219*100, it comes to 86%, not 84, and for acute, it is 717/1611*100 = 45, not 55%.
Then Figure 3, which has the same issue as Figure 1 for their instantiations. In addition, for both (a) and (b), it has numbers of pattern instantiations that do not match any of the numbers in table 1. Also, I can't figure out what's going on with the clusters: in the "generalization" in (a) it states that it is to be filled-up with cluster_103, cluster_143, and cluster_18, and then below the example instantiation in the "where"-part it shows what they are shortcuts for; except that 103 does not appear in that list in (a) and none of clusters 157, 3, and 35 appear in (b)'s list.
lexical irregularities: on "The detection of such irregularities is not feasible with the analysis of syntactic patterns …": in principle it could be feasible to find at least some of them, though, but one is expected to find more of them if the semantic regularities are included in the test. The end of the conclusion has a toned-down version, which is more accurate ("lexical irregularities in the entities were also revealed … after querying the ontology in question.")
section 6.2. "suggested methodology can work" -> method. On p4 it states [11] is used for detection of lexical patterns, but in the evaluation on p12, that [12] is used for that--either you have the same work presented in two papers, or there is a real discrepancy.
lexical patterns: why weren't all 464 processed?

Conclusions
"novel approach to unsupervised detection of semantic patterns in an ontology", for what evaluation has been presented, it is a case of 'finding instances of known patterns', and the unknowns from section 6.2 are still unknown to the reader.
"The qualitative analysis showed that semantic patterns provide a improved picture of an intended pattern and the deviations from that pattern.": to the authors, but I'm not convinced yet (though I'd like to be).

There are a few typos and font size issues as well, so if it gets accepted, do give it a thorough read and fix those issues.

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent

Novelty
Select your choice from the options below and write its number below.

== 5 excellent

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent

Clarity and presentation
Select your choice from the options below and write its number below.

== 4 good

Review
Please provide your textual review here.

Relevance to EKAW is very good: patterns of knowledge have been a longstanding mainstay interest of this community. This paper contributes to that tradition.

The paper builds on earlier work by extending syntactic/lexical ontology patters (of which there have been much work) with semantic patterns (on which there has been almost none).

The proposed method to detect such semantic patterns is fairly innovative (roughly: take the semantic closure of an ontology, do clustering on the resulting set of consequences, and use those clusters to identify patterns.

There is a substantial evaluation on a realistic dataset (SNOMED).

In summary: an exemplary paper. One of those papers where I thought "I wish I'd written that"...

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

1

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

4

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

3

Review
Please provide your textual review here.

The paper essentially extends the work on ontology regularity analysis, previously presented at ISWC'11 and in two biomedical journals. The new contributions are: 1) leveraging on entailed rather than just on asserted statements; 2) proposal for workflow; 3) presentation of a use case on SNOMED (for the new, entailment-based setting - the research actually seems to have been motivated by the shortcomings of the syntactic one when applied on SNOMED).

The principles of the approach appear sound, and the evaluation is convincing (set aside the fact that it is based on a single resource). The critical comments below thus mainly address the presentation aspects. Aside superficial issues related to language, I noted what seems to me as certain sloppiness in the formalization. What I particular miss from the reader viewpoint, is the lack of examples (given the approach is relatively complex). In particular:
- in Sect. 1 there should be a motivating example demonstrating how an intended pattern cannot be revealed by syntactic analysis.
- In Sect. 3 there might be an example of a multi-word lexical pattern.
- Section 5, referring to the principles of regularity detection, is almost unreadable without looking back to the ISWC'11 paper. A running example would help a lot, while some technical detailes could be spared.

Although this is probably quite common in the community where the authors reside, I do not find ideal to implicitly reduce the notion of 'semantic' (in contrast of 'syntactic') to 'based on logical reasoning'. 'Semantics' may also be understood as 'meaning', and a large part of the semantic web community considers this view legitimate. I suggest that you make your option for the meaning of this word more explicit, since 'semantic' could also mean, in the context of the paper, 'analyzing the labels relying on external thesauri or lexical ontologies rather than on literal strings'.

Detailed comments:
- Example in Fig. 1 should be explained contentwise. E.g., the property 'RoleGroup' is not intuitive. Same for Fig. 3 later.
- Definition 1 is probably too lenient, as it only describes a 'completion graph' by its structure and does not mention that it is related to some DL theory. It is also unclear there how can a single edge map to a *set* of properties. Instead, from the text below the definition it seems that there is only one such property. Finally, the mapping function L, according to the definition, should be applied on edges and not on pairs of nodes, as in L(y,x); even if multiple relations could indeed be packed into a single edge, it is not explicitly stated that there cannot be multiple edges between the same pair of nodes.
- Definition 2 should also include "of O", as the graph is derived from an ontology. Sig (O) should also be explained there.
- Sections 3 and 4 use the term 'label' in different meaning. The reader should be alerted on that.
- "In the future whenever we refer to KE we refer to the exploration of the graph (the letter `G' is dropped)." I find this unfortunate. A tool is different from the data it produces. Why would the extra letter harm?
- The notion of 'non-simple property' should probably be explained.
- Section 6: you only picked 308 cases out of 464. Why? (Just did not have time for more?) How were these chosen?
- The literature overview might also include the work by Allan Third, who also compared the parsed multi-word labels with axiom structure (the INLG 2012 "hidden semantics" paper; omitting axioms if covered by lexicalization, i.e. a bit opposite sense to what the authors propose). Generally, the number of cited works is relatively small, many of them are co-authored by the paper authors, and other are only marginally related; for example, [3] is not explicitly about ontologies.
- 'Citeseer' (for bibitem 15) is not a standard part of a reference. Afaik, WOP workshops use to have a CEUR volume?

Typos:
- Check the grammatical consistency, e.g., "patters... are not fully revealed", "Every edge... of its starting nodes" (it is a directed graph, i.e. there can be only one starting node?)
- "the detection of structural but the process"
- "methods for mining indefine methods for mining patterns"
- It seems that "OWLAPI OWLReasoner", with the second part in smaller font, at the start of Sect.4, is also aka typo?
- Alternating "Sig" vs. "sig" for the signature symbol.
- Alternating "placeholder" vs. "place-holder".
- "Table2"
- Bibtex artifacts: sparql, snomed.