OOPS! (OntOlogy Pitfall Scanner!): supporting ontology evaluation on-line

Tracking #: 989-2200

Maria Poveda
Mari Carmen Suárez-Figueroa
Miguel Ángel García-Delgado
Asunción Gómez-Pérez

Responsible editor: 
Marta Sabou

Submission type: 
Tool/System Report
This paper presents a system report about OOPS! (OntOlogy Pitfall Scanner!), a tool for detecting pitfalls in ontologies and targeted at newcomers and domain experts unfamiliar with description logics and ontology implementation languages. OOPS! extends the list of pitfalls detected by most recent and available approaches and allows selecting subset of pitfalls to be analysed according to different evaluation dimensions. In addition, the system also provides an indicator (critical, important, minor) for each pitfall according to their possible negative consequences. The system, which is free and online available, operates independently of any ontology development platform. Two interfaces are provided in order to make use of the system, namely 1) a web user interface supporting human-interaction with the system and 2) a RESTful web service supporting other systems to access OOPS! evaluation results. Currently, OOPS! has been used by developers from more than 50 countries (>2000 times), embedded within 3 third-party software developments and used in several enterprises, supporting both ontology development processes and training activities.
Full PDF Version: 

Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Silvio Peroni submitted on 05/Mar/2015
Major Revision
Review Comment:

This paper introduces OOPS!, a web-based tool to evaluate OWL ontologies according to a catalogue of "bad practices" (i.e., pitfalls) that may occur when developing ontologies.

As the authors explicitly say, this is not the first paper dedicated to OOPS! but, contrary to their previous works, it focuses on overviewing the whole system from technological and practical perspectives - which is totally fine with 'Tools and Systems Report' submissions.

# Summary

I firmly believe that this paper is a valid contribution to the SWJ and that should be published in this venue. However, I think that several issues should be addressed before deserving publication, in particular those referring to specific criteria proper to this kind of submissions, e.g., "clarity", "impact of the tool" and "limitations".

Please find my suggestions and concerns as follows.

# "Ontologies" and "OWL ontologies"

Reading the abstract (and other parts of the paper), it seems that OOPS! is able to analyse and evaluate any ontology, despite of the particular format used for developing such ontology and its actual expressiveness. For instance in the abstract the authors refer to "ontology implementation languages", in the introduction we have "an online service intended to help ontology developers", in section 2.1 "takes as input the ontology".

While I totally agree that the 40 pitfalls of the catalogue can be applicable to any kind of DL ontology, OOPS! works explicitly (and, as far I have understood, only) with OWL ontologies. This point should be clearly stated and clarified within the paper.

However, from the description of the "important" pitfalls provided in section 2, it seems that some pitfalls are referring explicitly to OWL, e.g., P25 talks about a constructor that the particular ontological language (i.e., OWL) can provide. Thus, are there some pitfalls that are actually OWL-specific only?

# Previous works on OOPS!

The part about authors' previous papers in the introduction (i.e., "While our previous papers [...] point of view of the system") should be introduced before the paragraph "The rest of the paper [...]" and should be expanded a bit in order to clearly understand what was analysed in the previous papers and what is the actually real new contribution in this article. In particular, I think "research and analysis work" is a bit vague here.

# Weak points of existing tools

The weak points of existing tools presented in the introduction do not seem so "weak" to me. In particular, why having a tool developed as a plugin of an existing system should be a negative point? The authors refer to the fact that there could be a misalignment between the earliest version of the main tool and the related plugin. However, I think this kind of scenarios can happen also in the context of web-based tools. What if I have a new (or old) version of a browser and OOPS! is not behaving correctly on it? Is it not the same issue?

The issue (b), the one about the installation of a Wiki technology, is not actually an issue of the Wiki itself, rather a problem of any kind of software that needs to be installed.

Finally, in the issue (c) the authors say that the "pitfalls" arisen by analising existing ontology evaluation tools are comparatively limited. Comparatively with whom - consider that the authors introduce OOPS! for the very first time in the following paragraph; maybe they should think to reword this part.

Summarising: the authors should extend the text here to explain and better support their claims.

# Limitations of OOPS!

In section 2.1 (and in other parts of the paper) there is an explicit reference to the fact that OOPS! handles only 32 pitfalls out of 40. This point is a limitation of the current implementation of the system. I would like to suggest to collect it with other existing limitations (e.g., the fact that currently it is able to accept only RDF/XML sources) in an appropriate section and then discussing possible ways to address them in future implementations of the system, even suggesting possible (and specific) approaches for dealing with them recognition.

In particular, for the recognition of the 8 remaining pitfalls, it is not enough to say that NLP tools will be used - I would like to understand which particular NLP tools the authors think are useful to address that limitation.

# About pitfalls

I understand that this paper is not about pitfalls and it focuses on OOPS!. However the 4th footnote states that some of the pitfalls recognised by the system may not be real errors, depending on the particular requirements the ontology engineer wants/is obliged to follow. I was wondering if some of these pitfalls that are not context-dependent and, thus, are always considered as errors whatever is the scenario one can consider.

# About the input via REST

In section 4 the authors present the fields of a specific XML document to send (via POST) to the service for analysing the related ontology. An example of such XML should be added within the text.

In addition, in the field "OntologyContent" the authors say that it is used to specify the source code of the ontology to analyse in RDF. Does it means that (even in the future) it won't be possible to use the Manchester Syntax or OWL/XML (that are not RDF-based syntaxes but still allow one to define OWL ontologies)?

# More evidence of use

In section 4.1 and section 6 there are specified some evidences that should demonstrate the actual impact of the system within the community, e.g., the fact that OOPS! has been used in existing project, and the fact that it has been used 2000 times from 50 different countries. I would like to see a (extensive) improvement of Section 4 in order to include, at least:

* a detailed indication (for instance, month by month) of when and how the service has been used, since when, what was the country that used it more (e.g., by looking at the logs in the server);

* how many different ontologies have been checked - in addition, the authors can consider to reveal the first ten;

* a detailed overview of other existing (and published) works that used OOPS! for validating ontologies, such as:


Note that I've just looked for them on Google Scholar in 5 mins, but a better job could be done with a more appropriate analysis of the existing literature.

# Comparison table

A table comparing all the other ontology evaluation tools with OOPS! according to specific functional points should be added. As an inspiration for such analysis, you can consider the (similar) table in

Peroni, S., Shotton, D., Vitali, F. (2013). Tools for the automatic generation of ontology documentation: a task-based evaluation. In International Journal on Semantic Web and Information Systems, 9 (1): 21-44. Hershey, Philadelphia, USA: IGI Global. DOI: 10.4018/jswis.2013010102
Preprint: http://speroni.web.cs.unibo.it/publications/peroni-2013-tools-automatic-...

# Interface issues

When I select "advanced evaluation" and then "Select Pitfalls for Evaluation" a list of pitfall ids is showed with no clear explanation of what they refer to. I see that a tooltip is poped up when hovering the cursor on them, but to understand their actual meaning I should hover each id, one by one. I would like to suggest to change a bit the approach here, writing explicitly (after the pitfall id) the actual label, and putting a more extensive description of the pitfall in the tooltip.

# Minor issues

- A general consideration: I actually think OOPS! is useful to any ontology developer, from newbies to experts. Anyone can make a mistake, after all.

- In the introduction, XD-Analyzer and Radon should be accompanied by appropriate links.

- In Section 2, the authors say that 32 out of 40 pitfalls are automatically handled by OOPS!, but there is not clue what these 32 are. Maybe a reference to Figure 1 (as the authors do at the end of Section 2) could be appropriate here.

- Figure 1 is introduced the first time at the end of Section 2, and I expected to find it on the next page.

- Is the bug about the RDF/XML as only possible input fixed in the earliest version of the system?

- In Figure 1 (printed in black&white) it is not clear the difference between important pitfalls and the others.

# Typos

intended to help -> for helping

bibliographical references, when possible -> bibliographic references when possible

how critical the pitfalls are -> the criticality of the pitfalls


Review #2
By Vojtěch Svátek submitted on 07/Apr/2015
Review Comment:

The paper describes an implemented tool for detecting potential issues in an ontology.

By the comparison with related research, the novel contribution of OOPS! seems to be:
1) larger coverage (but this requires proving that the additional "pitfalls" are truly frequent)
2) implementation as web app (however, the authors themselves mention that implementation as plugin "would be very convenient for the users since they would not need to change platforms to repair their ontologies after the diagnosis phase" - which is a bit of contradiction).

Sadly, the website http://oops.linkeddata.es/ was either completely down or crashed immediately when I wanted to try it (April 7). I however successfully tried an earlier version, and a student of mine tried it recently, so I know have no doubts the tool is real and works.

The authors say that "While our previous papers [15] and [16] have focused on the research and analysis work behind OOPS!, this paper gives an overview of the technological and practical point of view of the system." However, from the abstract of [15] I do not have the impression that the tool itself is uncovered there and warrants an additional journal publication. This however depends on the Editorial Board policy regarding 'Tools and Systems Reports'.
In any case, it is surprising that:
1) The current paper does not mention, in a digest form, the user satisfaction and pitfall coverage metrics from [15], as this is topical as evidence for tool quality.
2) The interface descriptions are so laconic and do not present more of the 'practical usage' of the system. Only the results for a query are shown, but not the ontology for which these results have been obtained.
I would not say that the size limit is already used up, and some space would be saved by only making the comparison with similar tools once, and not both in Sect. 1 and 5.

What is perhaps not in the IJSWIS article (at least, by its abstract?) and I would love to see is sample-based, manually evaluated detection accuracy for relevant 'tougher' pitfalls (e.g., whether 'wrong inverse relationships' are really wrong) and the overall statistics of individual detected pitfalls in terms of number of *externally*-submitted ontologies in which they occurred.

Section 2.1 mentions "suggestions" provided by a "suggestion scanner", without saying what these "suggestion" means (the term is not used elsewhere in the paper!).

In Fig. 2 it is not clear if the Evaluation results are provided to the user (as the users' and machines' icons are in a different part of the diagram).

Fig. 4 (with round 'radio buttons') evokes that the choices for individual dimensions are mutually exclusive. Is it really the case? If yes, then why?

As ideas for future work: the conventions checked should ideally be tuneable for a particular model style, such as 'LD vocabulary', 'webpage mark-up vocabulary', 'taxonomic-inferential ontology', or the like.

In the Intro, concepts/acronym are used without being introduced (e.g., LD) and tools/projects without or prior to citations/links (XD-Analyzer, Radon, Ontocheck, Moki).

As for related research, I am a bit surprised not to see this reference, as an author is from the same group:
Catherine Roussey, Óscar Corcho, Ondrej Sváb-Zamazal, François Scharffe, Stephan Bernard:
SPARQL-DL queries for Antipattern Detection. WOP 2012

English is rather bad in places. Ill-formed sentences, grammatical mismatches, typos, etc. Careful proof-reading is a must.
Also other typographic issues: font encoding issue in Section 4 and in biblio (no.9); (bibtex-incurred) decapitalized names in biblio.
Some parts of the text are unnecessarily verbose (contain superfluous elements), such as "OOPS! has been broadly accepted by a high number of users worldwide and has been executed more than 2000 times from 50 different countries. It has been continuously used from very different geographical locations."

Assessment summary in terms of the reviewing guidelines for 'Tools and Systems Report':
(1) Quality, importance, and impact of the described tool or system:
- quality of the system is hard to judge due to insufficient evaluation
- importance is ultimate, this kind of system is potentially very useful
- impact is, again, hard to judge due to insufficient evaluation
(2) Clarity, illustration, and readability of the describing paper:
- generally clear; however, missing account of tool's limitations
- illustration present but insufficient; only results but not input shown
- readability OK, but quality of English sub-optimal.
Additional issue not mentioned in the guidelines is unclear novelty with respect to a prior publication.

Currently I stick to rejection because of the likely lack of novelty compared to own published work: [15] (and possibly [16]).
If the event that the delta wrt. these prior publications were judged sufficient by the EB, I would see the revision to acceptable form as doable in relatively short timespan (say, 2 months), provided the evaluation provided promising results. However, major revision would be needed even then.

Review #3
Anonymous submitted on 15/May/2015
Major Revision
Review Comment:

The paper describes a tool called OOPS!, which can be used to detect common pitfalls in a given ontology. The pitfalls itself are provided by a pitfall catalogue, which was proposed in previous work.

This manuscript was submitted as 'Tools and Systems Report' and should be reviewed along the following dimensions:


(1) Quality, importance, and impact of the described tool or system (convincing evidence must be provided).

I agree that the tool
Technically, I would say that the quality of the tool itself is fair, it is more or less a web service which does what it was designed for. The web interface on the other hand could be improved in many directions, especially form a usability point of view.

Some comments according to the tool itself:
* The pitfall descriptions are only visible by (1) tooltip or (2) on some other page, which is not very comfortable and a user would have to know the meaning of each pitfall in advance or have it e.g on a separate sheet of paper. I would suggest to directly link from each pitfall to it’s detailed explanation in the documentation.
* I see encoding issues everywhere, e.g. int the tooltip for P03 or in the documentation of P18( “isOfficialLanguageâ€) - I'm using Linux (Ubuntu 14.10) + Google Chrome (Version 42.0.2311.152 (64-bit))
* there is no support of any kind of sorting of results e.g. by category


(2) Clarity, illustration, and readability of the describing paper, which shall convey to the reader both the capabilities and the limitations of the tool.

The paper is written in a good style and the structure is clear and easily to follow. On the other hand, there are some issues and some points remain open or are not discussed enough:

* It’s fair to mention weaknesses of plugin based tools, but the authors forgot to say that there are indeed limitations of online-services, e.g. that if the REST service is not available nobody can use OOPS!

* Is there any way to download OOPS! such that a user can deploy it locally on it's own machine? Moreover, is it supposed to be an open-source project and if so how can somebody get the source code?

* What are the limitations in size and complexity of the ontology? What happens if somebody tries to analyze a large ontologies like e.g. SNOMED CT (> 220,000 axioms) or even NCBI(>800,000 axioms)?

* Are reasoning services used? IF so, I guess OOPS! is somehow limited by the performance of a reasoner? If so, which reasoner is used and why? Are the any experiments that show how well it scales?

* How does OOPS! search for the pitfalls? By means of SPARQL queries or even JENA datastructures and API methods?

* How did the authors measure the usage? Counting IP adresses from the Web services calls? If so, which period did they consider?

* Can you give more details on how OOPS! was sued in the mentioned enterprises?


Minor comments:
* the JENA link is quite old, is already an Apache project called Apache JENA API and available at https://jena.apache.org/

Overall, I think that the tool is interesting and has some potential to help and support ontology developers, but because there are still too many open questions, I'd suggest to make a MAJOR REVISION first.