Formality and Accessibility in Ontology Representation and Reasoning: A Diagrammatic Approach

The paper presents an approach for visually explain inferences using Concept Diagrams (CD), which is an existing language for specifying ontologies. The paper also describes iCon, which is the tool that supports such visualisation approach. The authors included several appendix sections that include the CD language, the rules used to support the presented approach and additional explanation related to the use cases and the evaluation.

The paper is well-written and I also find useful the inference visualisation. However, I think that the current version of the manuscript still needs some improvements and explanations:

- At the beginning of the introduction section (Section 1), the authors describe ontology visualisation and expose some existing tools that deal with this topic. From that first paragraph, I would expect iCon to be an ontology visualisation tool that also supports reasoning. However, if I understood correctly after reading the manuscript, the proposed approach is not oriented to ontology visualisation but to explain the inference related to one ontology axiom. Therefore, it cannot be used for visualising the whole ontology.

- I also think that the authors should expose more clearly what is the contribution of the paper since the CD language was already presented in a previous paper.

- There is a lot of information that is only included in the Appendix sections, which hinders the readability of the paper. Could authors consider moving some of this information to the main sections? For example, in section 4.2 (page 6) there is no information about the inferences rules since they are all presented in Appendix B.

- Section 7.1. Are there any differences between the results obtained from experts in ontology development and those that are not experts? I think that this information is useful to measure the usability of the approach. It is useful for all kind of users? or only for those that are experts in logics and ontologies?

- In the Related Work section (Section 8) the authors do not mention Protégé plugins like VOWL or OWLAx which, since they are integrated into an ontology editor, also supports somehow the reasoning task (related to the claims stated in Section 1) although it is true that Protégé does not use the visual notation for explaining inferences.

- In the conclusions section, the authors state that "we proposed a unified approach for ontology representation, specification and reasoning.". However, it is unclear for me how to represent and specify the ontology with the reasoning approach proposed in this paper. If I understood correctly, the contribution of this paper is the visualisation of inference rules for ontology axioms.

The article attacks a crucial, central and timely research question (accessible knowledge representation & reasoning) by combining existing approaches (formal diagrammatic languages, diagrammatic reasoning, visual representation of proofs). I was very enthusiastic about the article's angle at first, as it proposes an ambitious and innovative research direction. The topic ideally fits SWJ.

The core of the presented results is an implementation and empirical evaluation of an interactive theorem prover for knowledge bases/ontologies. The underlying theoretical framework extends existing work on Concept Diagrams (CD) and the reasoning framework builds on and extends the existing Speedith framework (originally for Spider Diagrams, a "subset" of CD). The work is thus original but does not meet the level of innovation and ambition supposedly required to tackle the raised underlying research questions.

The focus and research horizon of the article is very confined. It focuses on a very limited subset of ontologies (OWL-ish ones or even OWL 2 RL) and a lot of the article's statements (viz. p19/48 [we are] "unifying diagrammatic representation and reasoning for ontologies for the first time") may only work in this very restricted setting. A proper embedding of the work in the field of diagrammatic knowledge representation and reasoning for ontologies (e.g. including the work of C.S. Peirce, Sowa's Knowledge Graphs and successors, Guizzardi et al.'s OntoUML,... -- all these proposed an unifying approach prior) is missing. This would help to better highlight the real original points of the article. A contrasting look on the currently rising field of explainable AI that tackles a similar question in a different domain, would also support the articles timeliness.

Even if the underlying research question and the proposed solution angle are certainly highly significant, the proposed results are not able to convince at the current stage.

This is mainly due to the missing clear direction of the article: the core concepts for evaluation (e.g. "accessibility") are never clearly defined (in a measurable way) and thus the proposed empirical evaluation is not traceable and transparently comprehensible.

For an empirical research article (as the authors would classify their paper themselves), additional details on the study and especially its participants, their background and their level of "accessibility" is missing. The proposed studies are not easily reproducible and seem artificial/academic but at least inspired/based on examples from real-world knowledge engineering. The evaluation mainly focuses on the proposed representation of micro-step diagrammatic proofs - ignoring the obvious rival of "explainable" micro-step symbolic proofs. Thus, the proposed supremacy of diagrammatic proofs in the given studies could be easily doubted.

For a concept/methodology paper (which is not really intended by the authors but would be necessary to make a clear distinction between the formalisms), formal proofs of the given statements (soundness etc.) are missing. Similarly the presentation of Concept Diagrams is neither sufficiently formal nor adequately intuitive for readers to follow the proposed argumentation.

Here, it would be desirable that the authors make a clear decision on WHAT they want to present and HOW they want to present it in a way that is self-contained and written with the SWJ audience in mind.

Thus, I opt for rejecting the paper in its current state. However, a substantial revision (i.e. a complete rewrite targeting the points raised above and which comprises certainly more than a major revision here) could proof a substantial scientific contribution worth publishing in the future.

***
Summary:
***

The paper "Formality and Accessibility in Ontology Representation and Reasoning: A Diagrammatic Approach" gives itself an ambitious goal, namely to "unify diagrammatic representation and reasoning for ontologies for the first time".

In line with such ambition, it covers an extremely wide spectrum of angles on diagrammatic reasoning for ontologies, namely describes a diagrammatic language (Section 2), reasoning for that language (Section 3), a description of a prototype implementation for that language (Section 4 and 5), a description of two different modelling case studies (Section 6), the description of an empirical study to evaluate how participants comprehend proofs given in the language (Section 7), and finally related and future work (Sections 8 and 9).

The study of diagrammatic languages indeed provides a welcome alternative angle to ontology representation and reasoning. In particular, one of the central topics of the paper, namely the use of diagrammatic representation and reasoning during debugging involving 'non-experts', seems indeed promising and worthwhile. It is also clear that the authors have done substantial previous work on the topic.

However, to sum up the more detailed comments below, the present paper provides an unbalanced presentation of too many topics and is not recommended for acceptance in this form. I would recommend the authors to reshape the paper to focus on the novel contributions and provide the necessary background in a more focused yet technically self-contained way. In short:

- the technical parts are on the one hand mostly not new (the CD formalism has been published previously), and what is presented in terms of technical material is sometimes ill-motivated and lacks discussion or detail.

- too many details are moved into the appendix, with an often underspecified reference/instructions. This has the effect that the main paper can appear partly incomprehensible (i.e. not self-contained) whereas the details provided in the appendix lack context and discussion.

- it appears that the only essentially new contribution is the user study carried out which is based on 10 participants inspecting only 4 hand-created proofs. Even though providing some interesting insights, the study seems rather limited in scope, particularly since it is carried out with 'logic experts'.

- typical advantages often discussed in the context of diagrammatic reasoning, such as intuitiveness, psychological/cognitive advantages, or 'free rides' in reasoning, are mentioned but not discussed in detail for the CD formalism.

- several of the presented aspects, such as the extend of coverage of the OWL 2 RL profile, seem rather preliminary. In particular, even though covering (fragments of) standard DL-based languages, no systematic comparison or translation of that standard syntax and semantics is given in the main text. This is particularly problematic for the typical Semantic Web Journal reader who is likely acquainted with DL but not with CD. Can a precise soundness and completeness result for a specific fragment be given?

***
Some Detailed Comments
***

Section 1: it would be nice to extend the introduction with some more background on diagrammatic traditions (Venn/Peirce/Euler etc) vs symbolic, and a more extended pitch why the diagrammatic might provide a valuable alternative and, on certain levels, something superior over the standard linear symbolic approach.

Section 2: the introduction of the language remains cryptic. An extended example (from the `crip sheet'?) would be useful to guide the reader to understand the formalism. The syntactic elements of the diagrams need to be more carefully introduced and their semantics explained. Ideally, a direct correlation to DL is given. For instance, saying that `Solid arrows mean that the source is related to only the target' is too vague as a definition, even when knowing that the `only' stems from DL talk. Another example: `Closed curves give rise to zones that are regions inside some or none of the curves and outside the remaining curves'. This is not sufficient as an introduction of the concept of 'closed curves'- are 'unclosed curves' admitted? Etc.

Section 3: you implement 24 out of 80 rules for the RL profile; the choice should be better motivated and the limitations explained. You equate `more granular' with `atomic' with `more likely to be explanatory' - where is the evidence for this? Tiny reasoning steps are not always easier. In fact, in many of the diagrammatic examples you give, an experienced reader can 'see' the inference immediately (eg Fig 4) whereas going through all the steps of the proof is tedious and not providing a `high-level' explanation.

Switching between Euler and Venn representations needs more motivation. Why is this kind of ambiguity not undesirable? Also, explain your naming schema such as `cax-dw'.

Section 4/5: it could be more immediate that this is an interactive system, otherwise ok as a summary of the iCon system.

In Section 5, we find a brief discussion of the one-to-many mapping of symbolic rules to sequences of diagrammatic rules. It remains somewhat hand-waiving. For instance, you say that there are 'infinitely many' valid inferences that can be constructed that do 'not resemble' an OWL 2 RL inference: what does this imply?

Section 6 discusses two use cases. It discusses how diagrammatic proofs can be given for certain relevant inferences. If the expressivity of the language is understood, it is rather clear that something like this can be done. I would consider this rather workshop paper material.

Section 7: I think the distinction between 'theorem proving' community and 'mathematicians' is rather misleading and wrong. Sequences and trees are used in both. The study seems useful to improve the design of the system, but rather limited to understand the general psychology and cognition involved in the formalism given the advanced knowledge of the participants. In terms of method, it is not clear what baseline would be used to measure the relative `accessibility' or `comprehensibility' if no alternative formalisation was provided.

Appendix: as commented before, some of the material should be in the main text, some other material would need to be enriched with discussion.

Part A contains a number of detailed technical definitions. It remains unclear a) which ones are novel, if any, b) which ones are needed in the main text, because they are not referenced in detail.

The central definitions come here without any discussion. Moreover, even though definitions such as Def 1 seem extremely detailed (having 12 parts), they are at the same time rather underspecified and lacking discussion. Are curves geometric objects or abstracts? Are shades just attributes of zones? A location is a set of zones? Why is the `equality' not transitive? What is the circle in part 8? Where do you define \mathcal{L}_S etc? Spider labels? Where have you introduced that distinction? Is Def 3 not exactly the same as a DL interpretation? Discuss that. And so on.

The 'crib sheet' (or parts of it) might be a good way to introduce the formalism also in the main paper, ideally with a symbolic translation to a standard formalism such as DL.

Typo:
to replace current abstract -> to replace the current abstract