Using ontologies - understanding the user experience

Tracking #: 733-1943

Authors: 
Paul Warren
Paul Mulholland
Trevor Collins
Enrico Motta

Responsible editor: 
Guest Editors EKAW 2014 Schlobach Janowicz

Submission type: 
Conference Style
Abstract: 
Drawing on 118 responses to a survey of ontology use, this paper describes the experiences of those who create and use ontologies. Responses to questions about language and tool use illustrate the dominant position of OWL and provide information about the OWL profiles and particular Description Logic features used. The survey revealed a considerable range of ontology sizes and analysis suggests a classification into two broad groups; one where ontologies have very few individuals, the other in which the number of individuals is more commensurate with the number of classes. The survey also reports on the use of ontology visualization software, finding that the importance of visualization to ontology users varies considerably. Pattern use is also examined in detail, drawing on further input from a follow-up study devoted exclusively to this topic. Evidence suggests that pattern creation and use are frequently informal processes and there is a need for improved tools. An analysis of the purposes for which ontologies are used suggests a classification into four categories of users: conceptualizers, integrators, searchers and multipurpose users. It is proposed that the categorisation of users and user behaviour should be taken into account when designing ontology tools and methodologies. This should enable rigorous, user-specific use cases.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
[EKAW] conference only accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
1

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

4
Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
3

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
3
Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
2
Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Review
Please provide your textual review here.

This paper is well written and contains an interesting analysis over the results of an ontology use survey. However, the conclusive findings are vague (basically described in the abstract) and it is unclear how they can be used.

Although the authors claim that the focus of the paper is on the user experience, there is no methodological definition of the user experience aspects they are trying to measure and improve.

A useful output could be a table relating types (categories) of users with desired tool/language features.

In Section 3, I would suggest to itemise the categories by name.

The last sentences in the abstract and conclusion suggest that this work is not complete. However, there is no description of what the next steps are.

What is the extended work planned for the journal version?

Review #2
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.
2
== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.
3
== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.
5
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.
3
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
4
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
4
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
4
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Review
Please provide your textual review here.

The paper investigates into the actual usage of ontology based on a survey with a significant number of participants. The statistics gathered and presented are the relevant ones and are the basis for the further development of semantic technologies in general. There are several works of that kind but the more there are, the better the situation is. I suggest to redo the same survey with similar participants in five years, since we do not know at what stage of dissemination of semantic technologies we are and whether the current usage is the initial usage or will be a stable one.

Review #3
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] reject
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

-1

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

3

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

3

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

2

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

Not applicable

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Review

Overall:
* The authors have tackled the laudable task of surveying ontology use. However: though survey seems to be easy to do, they are not. Constructing a survey is a difficult task and it seems to me that the authors have not done it successfully. Rather than just asking some question, the questions of a survey should be derived from an (or several) hypothesis, the questions should be developed in a way to ensure that inconsistent responses can be discovered, etc. etc. In the details I give some examples of what went wrong (though I am myself not an expert on constructing surveys I have to confess!).

* At the end of the paper I wondered what I had learned from the survey. The conclusion are ones that many people have without the survey (e.g. “one where ontologies have very few individuals, the other in which the number of individuals is more commensurate with the number of classes” is already given in Christoph Tempich, Raphael Volz: Towards a benchmark for Semantic Web reasoners - an analysis of the DAML ontology library. EON 2003 ). A scientific use of surveys is to confirm the wisdom commonly held and support it with numbers. Such a kind of number-foundation however needs to come with stronger hypotheses at the beginning.

* I am puzzled by the title. I had expected from the title that the paper was not about how large ontologies are and whether their engineering was helped by patterns, but rather how *users* experience them in a particular piece of software. This paper is rather about survey ontology engineering practice (though not quite). Note that again that seems to be due to a lack of theory and hypotheses that might have constituted a basis for the survey.

* I would suggest to have this as a poster at EKAW. It is definitely interesting for the community.

Details:
• I am puzzled by table 1, because what was the methodology to arrive at these category of uses? Why is there nothing about visualization? Why is there nothing about explanation/understanding (e.g. MagPie)? Maybe there is a rationale for this list, but it is not explained.
• I am puzzled by table 1, because some options seem to overlap and even contain each other, especially the questions concerning DI, LD and HD. This might actually be a possibility in order to determine inconsistent answers. E.g. DI should imply HD, should it not? If “no” what is the meaning of these explanations? If “yes” how does it come that DI is so much more prominent than HD? Furthermore, correlations are not sufficient to tackle these answers!
• A survey of 13 respondents, such as done in section 7.2 is not meaningful
• I do not understand Table 8. Does it mean that there are patterns with hundreds of classes? This does not seem to make sense. (maybe a pattern that is instantiated hundred of times? This would make more sense). Anyway the text does not tell

* Further References to be considered:
Birte Glimm, Aidan Hogan, Markus Krötzsch, Axel Polleres: OWL: Yet to arrive on the Web of Data? LDOW 2012, workshop at WWW-2012, CEUR-WS.org 2012