Development and Quality Evaluation of Hazelnut Ontology

Tracking #: 2410-3624

Authors: 
Sahin Aydin
Mehmet Nafiz Aydin

Responsible editor: 
Stefano Borgo

Submission type: 
Ontology Description
Abstract: 
In recent years, several projects that are supported by information and communications technologies (ICT) have been developed in the agricultural domain to promote more precise agricultural activities. Agriculture domain has a great deal of stakeholders. These stakeholders need more sophisticated data and appropriate intelligence to perform precise agricultural activities. It is essential to provide publishing domain-specific vocabularies while gathering data from heterogeneous sources and performing to merge them. When the importance of hazelnut agricultural product is taken into consideration, gathering much more detailed data regarding it and publishing this data for stakeholders of the relevant domain to use are indispensable. There is, therefore, a definite need for developing an ontology regarding hazelnut. In this research, we propose an ontology for hazelnut and examine a variety of ontology evaluation tools and methodologies to assess the ontology developed. In particular, we use a number of the metrics to evaluate the quality of proposed ontology and discuss the implications of proposed hazelnut ontology and its quality for both researchers and practitioners.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 25/Apr/2020
Suggestion:
Reject
Review Comment:

This manuscript presents various ways to assess Hazelnut Ontology. Authors refer to and take advantage of many well-known methodologies and evaluation tools. Unfortunately, the paper does not go beyond showing the results obtained from different evaluation tools. Therefore, no new insights can be drawn from the article.

It should be emphasized that no reference to the ontology was given, so the results presented in the paper could not be verified.

The evaluation presented in the paper mainly concerns the structural aspects of ontology. There is no discussion of how complete the ontology is, how many gaps it has, and how it behaves in the applications that the authors sketch at the beginning of their work. SMEs were not involved in creating the ontology in question; what from my experience may indicate that the ontology has no practical value.

Summarizing, in my opinion, the content and intuitions contained in the paper do not make it worth publishing in SWJ.

Review #2
By Pier Luigi Buttigieg submitted on 03/May/2020
Suggestion:
Reject
Review Comment:

(1) Quality and relevance of the described ontology: I cannot fully evaluate the ontology as the resource is not available (automatic grounds for rejection in most cases). From what is described in the manuscript, I do not see any convincing ontological work or progress here, simply a rendition of content present in a single document. In terms of relevance, hazelnut ontologies would be welcome to extend the field of agronomic ontology; unfortunately, I cannot see such content here. More detailed comments below.

(2) Illustration, clarity and readability of the describing paper, which shall convey to the reader the key aspects of the described ontology: The paper is readable, but lacks focus and frequently pursues tangents that do not contribute to its aims.

Comments below are provided for the authors' consideration, should they choose to resubmit.

General comments

* The actual ontology artifact should be made available and the IRI to it clearly displayed in the Abstract and introduction.

* the submission must be thoroughly revised for grammatical correctness and formal language. The errors and use of colloquial language are numerous enough to decrease readability and also lead to ambiguity.

* There are many statements in this manuscript that require at least some citations. The opening line of the introduction is one of them.

* The use of the terms "stakeholder" and "domain expert" is, at times, very loose - the authors should define these or substitute them where they mean something more specific.

Abstract

* I'm not sure what "provide publishing domain-specific vocabularies" means

* "When the importance of hazelnut agricultural product is taken into consideration, gathering much more detailed data regarding it and publishing this data for stakeholders of the relevant domain to use are indispensable." Why is this so? Some supporting arguments for claims such as this one would be useful.

Introduction

The "ICT" umbrella is massive. The authors must be more precise when introducing what components of ICT are relevant to this work or the hazelnut industry (with references).

The introduction is full of redundant statements and weakly made claims with no referencing. The reader is not provided with any perspective on the value of the hazelnut industry or how ICT has impacted it to date. This section should be dramatically condensed and be more focused.

"Ph" --> "pH"
Further, listing parameters with no overarching introduction on why these are important (with references) may actually be more distracting than helpful.

The interoperabiliy aspect of domain-specific vocabularies is not discussed. Note that it does not matter how many vocabularies are published if they are not interoperable with others - this only adds to the heterogeneity the authors identify as an impediment to progress.

"Ontologies enable us to share a common understanding of the structure of information among people or software agents,"
This point is not a natural function of ontologies. Only very specific ontologies focus on information structuring.

What are "semantic systems"? In general, the introduction tends to throw terms and concepts at the reader that aren't clearly described or relevant to the main thrust. This makes it hard to see where the authors are going.

"A variety of tools and methodologies are used while evaluating the ontologies to eliminate these problems."
I'm afraid this is another example of a claim without any references or backing. Please develop this if it is to be retained (see other comments on the need for a more focused scope)

"to evaluate the proposed ontology by using different tools and methodologies."
Evaluate it for what? Against what criteria and why? Details can be listed in the methods, but the reasoning behind this evaluation must be made clear in the Introduction.

"Furthermore, the proposed ontology might be used as the main component of semantic annotation layer within the scope of multi-layer agricultural open data processing model [6]."
How is this a furtherance of the arguments that precede it? The logical flow here is questionable. Also, why "might"? The reader is left with many questions.

"This research makes noteworthy contributions to the current literature in two ways"
It's somewhat forward to claim the research as noteworthy, a priori.

"First, we review the research conducted on existing agricultural ontologies; then we propose an ontology for hazelnut."
This manuscript has something of an identity crisis: is it a review or an ontology description and evaluation? The paragraph continues to list more goals which seem arbitrary.

"The paper has been divided into four parts..."
This is much more clear - the authors should constrain themselves to a clear problem. However, this is still not an introduction, just a table of contents in prose.

Research Background

"Eliminating heterogeneity of data sources in the agricultural domain is possible by using semantic web technologies."
This is a questionable statement that oversimplifies the challenges of handling heterogeneous data. Semantics can only help with one part of that problem.

"Semantic web targets ensure significant contents’ format, which could be processed by both humans and machines"
I'm afraid it's not clear to me what this means. The rest of the paragraph clarifies this to a degree, but many topic sentences in this document are puzzling and should be revised.

Given that this is the SWJ, I think the section on RDF/OWL/etc is not needed. Please focus on the ontology developed here and simply explain why you chose the encoding you did. The same is true for the general introduction to what an ontology is.

"they are also implied in projects as conceptual models"
This isn't so: a conceptual model is not an ontology (they overlap to a degree, but it's useful to be more precise here) and can't do the tasks that follow in the prose. Also, "implied" is perhaps the wrong word.

AGROVOC is more a thesaurus with SKOS-level expression, rather than an ontology.

The listing of imported ontologies in the CO is poorly reported. This section must be revised for accuracy. Many are not "ontologies of the CO", just interoperating ontologies from the OBO Library.

"However, the arguments proposed by domain experts have proved that any ontology is not appropriate for hazelnut."
Where are these arguments? These are likely to be useful to the readership and would be interesting to list systematically.

Table 1 seems like the beginning of something interesting, but it's underdeveloped and its purpose is unclear: Why would only the CO be examined for the content needed to create a hazelnut ontology? Many of the dimensions identified as lacking (e.g. passport, management environment) wouldn't be found in the CO but in other (e.g. OBO) ontologies (or at least superclasses that could be reused). Thus, this is somewhat of a strawman argument for motivating the creation of a brand new ontology, largely ignoring interoperability and reuse.

It's not obligatory to reuse other ontologies for research or applications, but if that was the intention, it's best not to claim that existing resources weren't "sophisticated" enough for a use case without presenting a fair or systematic evaluation. It would be more credible to say that a new, independent ontology was created to meet a highly bespoke set of needs, and that reuse or interoperability were not priorities here.

3. Model and Findings

This section begins with another round of introductory material that should be moved to the Introduction.

"There is, therefore, a definite need for developing an ontology regarding hazelnut."
This is likely, but the authors have yet again made a claim with no specific reasoning or references to articulate this need.

"On the other hand, Hazelnut Ontology is acknowledged that it is going to help provide an international format to standardize general understanding with respect to hazelnut."
I don't find this supportable, when no real effort to interoperate with existing global resources was (apparently) made. I would have to examine the actual ontology to verify this, but that doesn't seem to be available.

"We will examine a number of methodologies adopting the approaches for the purpose of assessing the Hazelnut Ontology in detail."
I understand the motivation for doing this, but this form of paper is far too concise to give a reasonable assessment of these approaches. It's acceptable to simply choose one, stating the general rationale behind the choice. The main issue is interoperability.

The listing of various upper-level / development frameworks doesn't seem to go anywhere. Again, the manuscript should focus on the ontology actually developed here rather than going on tangents.

"The main purpose of this study is to develop an agricultural domain ontology and evaluate the quality of this ontology. That’s why, for the first step of the relevant methodology, agriculture has been determined as the domain of creating ontology. Furthermore, this study aims to contribute to Hazelnut Farming area of research by developing an ontology."
This is an example of redundant repetition, likely added as the tangents in the paper would distract the reader from the purpose. Please stay on purpose.

"As the existing ontologies are not appropriate for hazelnut agricultural product, we could not make use of these existing ontologies while developing the ontology."
I find this impossible to believe: there are no ontologies that can any classes that are reusable? http://purl.obolibrary.org/obo/PO_0030102

'The most comprehensive document to determine the metadata regarding hazelnut is “Descriptors for Hazelnut”.'
A more careful analysis of this document as the source of the present ontology would make for a much more interesting and principled manuscript, I think. Additionally, what makes this the "most comprehensive" document available?

"These descriptors are classes of the ontology, and also, they designate the class hierarchy with the sub descriptors. Meanwhile, it should be noted that each descriptor has several sub descriptors. Some of these sub descriptors are classes of the ontology, but the others are the properties of classes-slot. The definitions and number of sub descriptors of each main descriptor are demonstrated on Table 2."
This is something of an issue - if the descriptors are simply extracted from the document, where's the actual ontological work? How does this ontology make the document content more intelligible to machine agents?

Table 2: The definitions here are not very semantically helpful. They describe what the entities "do", rather than what they are. This doesn't make for a quality ontology, I'm afraid.

"...so while creating Hazelnut Ontology, this hierarchy should be considered."
Yes and no - the hierarchy isn't necessarily an ontological one.

"Ontology creation is based on iterative design, and complicated and evolutionary process. There is no doubt that many domain experts shall contribute to Hazelnut Ontology."
This is sometimes true, but future work is not assured.

"We begin by taking a closer look at frameworks, methodologies and tools to evaluate the quality of Hazelnut Ontology."
Again, there is no scope here to provide a meaningful review of these tools. Please simply state why you chose them for evaluation. Section 3.2 should be mostly removed.

"As Hazelnut Ontology is a typical instance of an agricultural ontology, it is a kind of trait dictionary because it reflects detailed descriptors with respect to hazelnut agricultural product in five different groups."
This needs to be rephrased for clarity. What makes this ontology typical of the agricultural domain? What are comparing it to?

Section 3.3 again goes off on several tangents that do not focus on the evaluation of the Hazelnut Ontology itself. These segments should be removed.

"...are used to evaluate Hazelnut Ontology within the scope of this study"
It is unclear what the scope of this study is.

"When one looks at Table 2, one can see that each metric (class axioms, object property axioms, individual axioms, and annotation axioms) have sub-metrics as well."
Table 3. Further, many of these metrics are not relevant ot an evaluation of an ontology: most are simply descriptive.

This section goes on to read like excerpts from a Protege manual. How is this relevant to the evaluation of the ontology at hand?

The fact that there are no equivalence axioms concerns me.

I would much rather see demonstrations of the ontology's usage and effectiveness rather than a collection of tables that could mean many things. This section doesn't really evaluate the ontology against any meaningful test suite.

"Equivalence ratio enables us to calculate the rate between similar classes and all classes. It is computed as the number of the same classes divided by the number of all classes."
This must be rephrased / revised.

"Ontology quality can be measured considering the data that took part within the ontology as it points to how well the ontology is designed, and how much the ontology represents the real-world."
This is not really accurate - One can have thousands of instances, but still have a nonsense ontology. Testing the ontology through querying, reasoning, etc against a set of competence questions would be meaningful.

The rest of Section 3.3. is more of the same really, and doesn't really evaluate the ontology at all.

"It is not convenient to answer some of these questions due to the type of ontology."
There would have to be a far more principled account for this.

Section 3.4 is very strange and unconvincing. The grading methodology and process is not clear and the scores are suspiciously high, when the design of the ontology is essentially a hierarchy of descriptors taken from a document.