A Conceptual Model for Ontology Quality Assessment

Submission Tracking #: 2850-4064
Authors: Shyama Wilson, J. S. Goonetillake, W.A. Indika, Athula Ginige
Submission type: Survey Article

------- Overall Evaluation --------

Accepted with major review.

*** Summary Review ***

The purpose of the paper is to tackle the nonexistence of a universally agreed methodology for specifying and evaluating the quality of an ontology, by providing an analysis of existing contributions focusing on ontology quality models, characteristics, and the associated measures of these characteristics.

+ Precise paper structure
+ Useful summaries in Tables and Figures
+ Clear and well explained pipeline of selected works

- Lacks in grammar and syntax in the text, often in crucial parts of the paper
- Absence of some topic relevant works

*** Overall Evaluation ***

The paper is well organized and it follows a compelling narrative, moving section by section through the proposed conceptual model. Some really useful Tables and Figures are provided to clarify the structure and ripartition of the survey, the research problem is clearly stated and the research questions are specified as well as the methodology used in the selection of works treated in the survey.

The first main problem is the severe lack of a formal revision: there is almost no section, table and main paragraph without some formal inaccuracy. As non English native speaker I perfectly understand the eventuality of some typos in the text, but, as I tried to document in detail, if the inaccuracies are encountered too often in the text and in crucial parts (e.g. the body of the definitions) the risk is to compromise in some way the readability of the paper, or to leave the reader with the doubt of not having understood the very precise sense of the sentence. In a Survey Article providing 19 definitions the lack of syntactic accuracy needs to be filled, for this reason I strongly recommend to revise the whole work, since the typos listed in this review are probably not all those present right now in the text.

The second issue stems from Section 2, Survey Methodology: although the paper explicitly declares in detail the criterion of selection and focuses on a listed number of papers, the title and the broader purpose declared in the Abstract and Introduction of the present work aim at enlarging the perspective. With the purpose of providing a general Conceptual Model for Ontology Quality Assesment, at least to my knowledge and perspective, it seems necessary to mention Ontology Design Patterns, and in particular the volume "Ontology Engineering with Ontology Design Patterns: Foundations and Applications", specifically Karl Hammer's chapter "Quality of Content Ontology Design Patterns" focused on Content Ontology Design Patterns Quality Meta-model (the mentioned volume is in English, 2016, so it could be included in this Survey Article according to the Inclusion Criteria declared in Section 2. It is eventually debatable their being "directly relevant to the research questions", but in my opinion since the purpose of a good Survey Article is to include the highest quality introductory and overview texts, it should be at least mentioned their existence and the relevance of the approach to the matter).

*** Sections Review ***

--- Section 1 [Introduction] ---

"...overview of the exiting..." → "...overview of the existing..."

Also, Introduction and Figure 1 could benefit from mentioning ODP and considering Xtreme Design workflow while assessing the three bullet points resuming difficulties faced by researchers and practitioners (in particular bullet point 1).

--- Section 3 [Preliminaries and conceptualization] ---

"For instance cognitive complexity is one of the..." → is there a reference definition of "cognitive complexity"? if so, please provide it here, if not, try to better define what is meant here, or rephrase in order to specify the problematicity of a unique definition.

"The structural layer / architectural layer focuses on the is-a relationship which is more important in the ontology modeling against other relations." → I would recommend to be very cautious when asserting that something is "more important than": it is a more frequent relation in the sense that it gives a taxonomical order to entities in the ontology, but from the semantic perspective it is even less informative than any other more semantically determined relation.

--- Section 4 [Ontology Quality Models] ---

4.2 :
"It describes <> with..." is the definition taken from [80] as well as the following levels listed? if not, specify the source of the definition.

"The reason would be the proposed sub-characteristics are subjective and difficult to apply in practice" → I don't understand totally the meaning of the sentence: i think there is a problem of syntax. Is the "would" a critic or skeptic position of the authors? if not, rephrase in order to better explain the content please.

Table 2:
Quality model in ONTO-EVOAL Description: "the certain characteristics have been" → I think "certain" is not the proper word here.
OntoQualitas Description ends with a ","
Quality Model of Zhu et al. Description: "the five weather ontologies" → which five weather ontologies?

--- Section 5 [Classification of ontology quality characteristics] ---

About the definitions: the cleaner the better. Some of the sentences sound incorrect or redundant, in particular the definitions should be corrected and the overall form should be improved.
Some specifications:

5.1.1:
Definition 1:
"refers to the ontology is" → "refers to the ontology being..." or "refers to the fact that the ontology is..."

5.1.2:
Definition 2: as before the definition of "cognitive complexity" should be introduced at some point. From the cited references in bibliography the issue is faced at least in [80] and [5] (Cognitive Ergonomics).

5.1.3:
Definition 3:
Internal consistency refers to the ontology is → "refers to the ontology being..." or "refers to the fact that the ontology is..."
Also, this section could benefit from Ontology Design Pattern's chapter by Tiago Prince Sales and Giancarlo Guizzardi about Anti-patterns.

5.2.1:
"Identical formal definition of some classes: is appeared when..." → I don't think "is appeared" is the right form here. Also, maybe better specify that if there is an identical formal definition of some classes there could be a typization problem or incosistencies if there is also a disjointness among the same entities.
"Identical formal definition of some instances: is appeared when..." → same as before.

Definition 5:
"Thus, in an ontology, explicit redundancies do not exist between definitions, and redundancies cannot be inferred using other definitions and axioms." → maybe better "...explicit redundancies do not exist between definitions neither can be inferred using other definitions and axioms."

5.2.2
Definition 6:
"...in the domain that the ontology is being modeled." → "in the domain that the ontology is modeling/covering" or similar.

5.3.2 and 5.3.3
Definitions 9 and 10 seems really simlar, in case specify the difference between them or consider the possibility to collapse them, or at least determine a possible relation between them e.g. an ontology is relevant when it is funcionally complete.

5.3.4
Definition 11:
"...of an ontology and its elements that enable users (i.e., ontology consumers) to understand the..." → "of an ontology and how its elements enable users..." or similar.

5.4.2
"throughput" → "throughout"

5.5.1
The title "Currentess" → "Currentness".
Also, isn't Definition 14 a further elaboration of Definition 6 declined considering the time variable? The Currentness seems to be a specification of Coverage dimension considering a lack of coverage due to the changing of information and missing update of the ontology.

5.5.3
"the information is needed to provide on time for the..." → maybe "to be provided on time"...? I don't totally catch this nuance, due to this I don't totally understand Definition 16.

5.6.2
"There is no agreed definition is provided..." → "There is no agreement in the provided definition" or similar.

5.6.4
"...it is importan to consider because ontologies are evolving due to changes in the needs of the application, changes in the domain, changes in conceptualization, and changes in the explicit specification." → I think it is just lacking a pronoun: "...it is importan to consider it, because..." but "due to changings in...etc." sounds also better.

Table 9:
"Curretness" → "Currentness".
The measure of History attribute: "public library" → maybe better "open resource" ?
The measure "Average update rate" seems to better measure Currentness instead of Volatility.
I don't understand the measure for Availability, rephrase it in a proper way please.

5.7.
One of the main debate could be among the Characteristics and their relationships: albeit it is a Survey Article I was expecting some more discussion in this section, even without answers but only proposals of comparison.

5.8
"...has evaluated U ontology..." → I'm not sure of the meaning.

The authors present a system review on the topic of ontology evaluation. I think that the content of the paper fits well with the scope of the journal and complies with the evaluation criteria for systematic review papers. In addition, ontology evaluation is a challenging topic of research, and although different approaches and metrics have been provided, no standardized approach has emerged yet. From this perspective, I think that the research presented in the paper contributes to identifying existing criteria for ontology evaluation. Further work on the topic could contribute to bringing about a robust methodology for assessing the quality of ontologies, at least, according to certain quality criteria.

I must say that I have never conducted research on the methodologies/approaches presented by the authors. My evaluation of the paper takes therefore primarily into account the clarity and coherence of the paper with respect to broader research efforts in ontology engineering.

From this perspective, what I think is missing from this review is reference to methodologies for the conceptual analysis of ontologies. I have in mind a methodology like OntoClean, which has been embedded and further developed in the language OntoUML (see references below). An ontology is not only a logical theory about a domain; it is based on conceptual modeling principles, and these have to be taken into account when evaluating the quality of an ontology. Similarly, it is a good practice for ontology development to reuse existing top-level ontologies, where high level distinctions have been already drawn and formally specified (see Keet 2011). The authors need to consider the inclusion of these aspects into their analysis; or at least, they need to explain why these aspects are not included in their analysis.

Comments that I hope can help the authors to improve their paper:

- It seems to me that the authors have mainly in mind ontologies specified in Semantic Web languages (RDFS, OWL, etc.). This has to be clearly specified throughout the paper. E.g., when presenting papers in the state of the art (cf,. Table 1-2 etc.), it needs to be said which formal languages were taken into consideration. The selection of a representational language makes indeed a strong difference for ontology evaluation.

- For the inclusion criteria (section 2), the authors considered papers published between 2010-2021. They are, however, aware of important papers published before 2010 (e.g., by Gangemi et al.'s). Why were these papers excluded from the analysis? More generally, why does 2010 play such an important role for inclusion in the scope of this research?

- Figure 3: what are the elements of this graph that have to be necessarily taken into account? Characteristics and measures? It would be convenient to say that explicitly.

- End of section 4.2. When talking about top-down approaches for ontology evaluation, I don't understand what "ontological theories" means. Some clarifications are needed (e.g., Table 2, OQuaRe > approach: Top-down approach based on the ontological theories).

- Section 5. The authors classify the characteristics for measuring the quality of ontologies in four dimensions (see Table 4). This provides a clean and tidy classification. The problem that I see here is that some characteristics are borderline or, better said, they lay at the intersection between multiple dimensions (see the case of modularity below). I'm therefore wondering whether it would be better to place some characteristics across dimensions.

- Section 5.1.1. I understand that compliance refers to the syntactic correctness of the selected knowledge representation language. I find ambiguous reference to standards and conventions in definition 1. I would simply say something like: an ontology designed in language L is syntactically compliant with L when it is constructed according to the syntactic rules of L (e.g., first-order logic has clear rules for well-formed formulas).

- Section 5.1.3: I would recommend the authors to simply call 'logical consistency' the kind of consistency described in this section. This is indeed the common terminology used in computer science and mathematical logic. Also, I would suggest to reuse a standard definition of logical consistency as given in logic textbooks. Also, I don't understand how "circularity" and "partition" errors fit into this criteria; clarifications are needed.

- Section 5.1.4: modularity is classified as a structural intrinsic dimension of ontologies, hence, as a dimension that "does not depend on the knowledge of the domain". This entails that the choice of developing an ontology in a modular fashion is a purely formal matter. This is however controversial; as the authors themselves say (beginning of section 5.1.4), the choice of modularizing an ontology in a certain manner can depend on the representation of a specific topic, therefore it depends on domain knowledge. As said above, modularity may actually stand at the intersection between (at least) structural and domain intrinsic. For a review on modularization, see: Khan, Z. C., & Keet, C. M. (2015). An empirically-based framework for ontology modularisation. Applied Ontology, 10(3-4), 171-195.

- Section 5.2. I think that it is better to say "whether an ontology for a certain application domain is modeled according to the relative domain knowledge". Reference to "the real world" can be misleading. Consider the case of ontologies for fictions (cinema, literature, etc.): do the classes of these ontologies represent the "real world"?

- Section 5.2.2. It is not clear in which sense domain coverage can be detected via internal (aka logical) consistency. Assume that all objects of my application domain are either desks or chairs, and assume that I forgot to represent desks. Hence, the ontology is not complete, it would be affected (I assume) by an incompleteness error. How is it possible to spot this error via checking its logical consistency? Clarifications are required.

- Section 5.2.3. The authors confuse logical consistency with coherence wrt domain knowledge (external consistency). The definition given by Gomez-Perez is about logical consistency and definition 7 talks of being free from contradictions, which was already part of internal consistency! In addition, the quotation saying that a definition is consistent iff metaphysically consistent wrt the real-world does not add much to the discussion. It is actually confusing: what does "metaphysical consistency" mean? I would suggest removing this sentence or clarifying it. In any case, the authors must pay attention in distinguishing properly between logical consistency and domain coherence (external consistency).

- Section 5.3. I think that the term 'domain extrinsic' is unfortunate, because it suggests that these criteria do not depend on the domain, while at first glance they depend on it. E.g., in order to understand whether an ontology meets users' requirements via competency questions, or whether it provides information to accomplish users' tasks (Relevance), one must consider domain knowledge.

- Section 5.4. The same consideration as above. I find it confusing to call these characteristics 'application extrinsic' while they do depend on the application system.

- Section 5.4.1. I find Definition 12 confusing, because it talks of changing an ontology without modifying some of its portions. I think that the authors should better explain what they have in mind. There can be different manners in which an ontology may be adapted to certain application domains; e.g., by extending it via specific subclasses and relations, by weakening some of its axioms, etc.

- I'm wondering whether it is possible to individuate a minimal core of criteria that each developer must take into account to provide a valuable ontology. E.g., there may be essential quality criteria like logical consistency, optional criteria like modularity (after all an ontology does not necessarily need to be modularized), etc.

I'm not a native English speaker but I would recommend the authors to double-check the paper for typos or sentences that are not properly written. Some examples:

Section 1:
- this would enable ontology is to be used → this would enable an ontology to be used
the good quality ontology → a good quality ontology
- is available → were available (hypothetical statement)
- however not so far, as the way it is in software engineering -> this sentence requires to be rewritten.
- Therefore, these models do not guarantee work well for other contexts rather than its specified context -> Therefore, these models do not guarantee to work well for other contexts rather than their specified context.

5.1.3, later one → the latter one

There are other typos...

References:
Guarino, N., & Welty, C. A. (2004). An overview of OntoClean. Handbook on ontologies, 151-171.

Keet, C. M. (2011, May). The use of foundational ontologies in ontology development: an empirical assessment. In Extended Semantic Web Conference (pp. 321-335). Springer, Berlin, Heidelberg.

https://en.wikipedia.org/wiki/OntoUML

https://ontouml.readthedocs.io/en/latest/index.html

This paper presents a systematic survey for ontology quality assessment, focusing on on ontology quality models, characteristics, and the associated measures of these characteristics.

I applaud a systematic review in the area, as existing surveys are really limited and short-sighted.

Overall the paper is easy to read and well-structured, with clear contribution to the field using an appropriate methodology and filling an existing gap in the literature. It is appropriate for researchers, PHD students and practitioners to get started on the covered topic, it is quite comprehensive and covers existing state of the art.

=3. Survey Methodology

“However, they are not comprehensive surveys” I would rephrase this to better explain what is missing.

=3. Preliminaries

A table with definitions of the key terminology in the area would be highly beneficial for the paper.

=4. Ontology Quality Models

A preamble is required in the beginning of section 4 to explain why quality models in software engineering are placed here, and then in Section 4.1 specifically what can be (or were) reused and how.

I also find the generated tables really useful.

==5. Classification of ontology Quality characteristics

In table 4, the reader by just looking the figure cannot identify the difference between the dark rectangles and the white ones. An explanation is needed either in the figure or in the table legend.

Definition 2. Complexity is defined by using the term “complexity” again. A better definition is required here.

I see ontology evaluation as a continuous cycle, as an ontology is not a static artefact but subject to continuous change and adaption. This is an interesting dimension that should be incorporated in your survey in order to be complete along with some relevant references from ontology change (e.g. [x2, x3]. It could be added in the “adaptability” or some other characteristic as well.

==6. Conclusions and discussion

It would be interesting to discuss what would mean an ontology to be accompanied with its characteristics. Which of those characteristics you see as the most important? Would this increase ontology sharing and reuse?

The authors could also check [x3] another recent paper on the topic.

References
[x1] Ontology change: classification and survey, G Flouris, et al., The Knowledge Engineering Review 23 (2), 117-152
[x2] Ontology evolution: a process-centric survey, F Zablith, et al., The knowledge engineering review 30 (1), 45-75
[x3] Delta: A Modular Ontology Evaluation System, H Kondylakis, et al. Information 12 (8), 301

As a meta-review, I invite the authors to revise their paper following the major and minor suggestions of the three reviewers, especially considering approaches that are not covered: ontology design patterns (Reviewer1, and citation [2] below that delves deeply into the functional dimension), predicate attributes such as in OntoClean (Reviewer2), and formal assessment of local parametric goals as with O2 and oQual (citation [1] below).

As a personal note, although the authors make wide usage of papers that are companions to, or are inspired by, [1], [1] is not cited. The approach itself (semiotic metaontology of quality+O2 and oQual) is not included in the table summary at Appendix A. I also suggest to include design patterns-based and predicate attribute-based methods in that appendix.

[1] https://link.springer.com/content/pdf/10.1007/11762256_13.pdf
[2] https://dl.acm.org/doi/abs/10.1145/1597735.1597743