Adventure of Categories: Modeling the life-cycle of categories during scientific investigation

Tracking #: 731-1941

Authors: 
Prashant Gupta
Mark Gahegan
Gillian Dobbie

Responsible editor: 
Guest Editors EKAW 2014 Schlobach Janowicz

Submission type: 
Conference Style
Abstract: 
Categories are the fundamental components of scientific knowledge and are used in every phase of the scientific process. However, they are often in a state of flux, with new observations, discoveries and changes in our conceptual understanding leading to the birth and death of categories, drift in their iden-tities, as well as merging or splitting. Contemporary research tools rarely sup-port such changes in operationalized categories, neglecting the problem of cap-turing and utilizing the knowledge lurking behind the process of change. This paper presents a tool – AdvoCate – that represents the dynamic nature of cate-gories and allows them to be modelled and to evolve, while maintaining a cate-gory versioning system that captures all the different versions of a category along with the process of its evolution; this helps to better understand and communicate different versions of categories and the reasons and decisions be-hind any changes. We demonstrate the usefulness of AdvoCate using examples of category evolution from a land cover mapping exercise.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
[EKAW] reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 20/Aug/2014
Suggestion:
[EKAW] reject
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
-2

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

4
Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
2
Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
3
Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
1
Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
3
Review
Please provide your textual review here.

The paper presents a system to capture categories that may be defined during a scientific workflow. If I understand correctly, the presented system should be positioned somewhere between an ontology editor and a scientific workflow system as it doesn't seem to aid the user in making their exact experiments more reproducible, but rather make the ontology creation system more traceable. I think this could be a very helpful tool in collaborative knowledge capture efforts as it has the potential of helping collaborators understand the choices made.

The paper describes the overall system architecture in detail, as well as a sample scenario of a changing category structure. However, it is difficult to assess the tool as there is no link provided to a demo, and a Google search also did not yield an option to try out the tool. This is related to my main concern about the paper, namely that it is not clear what the exact status of the implementation of the system is. Furthermore, the paper does not include an evaluation of the system, or a mention of it in the future work section. Such a system could potentially be disruptive to the way we capture knowledge, but there is no mention of users consulted during the design process or testing first versions of the system. Perhaps the authors could elaborate on the context of the project, and its potential users (from the scenario given it seems that one target group is government services, are they involved in the project?). The paper also does not mention in what kind of format the data accrued by the tool will be stored (and how it could be used afterwards). Are they using any open standards here? (which would also aid potential uptake)

It is also not clear whether the images in Figure 5, present actual screen shots or whether these are mock-ups of the system. In its current state, the paper presents an interesting idea, but in order to accept it for this conference I think the paper needs the following:

- clarification of the implementation status
- description of target audience and their involvement
- evaluation of the system (both the algorithms in there, as well as the interface)

Minor remarks:
- sometimes articles are missing in noun phrases e.g. the title of Section 3 "Overview of AdvoCate Tool" would be better as "Overview of the AdvoCate Tool"
- It's not really necessary to start a new section after the Whitehead quote in the Introduction
- page 3: haven't -> have not

Review #2
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
1

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)
4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
3
Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
3
Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Review
Please provide your textual review here.
The paper presents a system able to support category structure evolution over time, claiming to enable to capture behind the process of changes. The concept is examplified using a land cover mapping exercise.
The issue is highly relevant and often neglected - and existing systems rarely address all the relevant aspects, which include category versioning, tools to support change management and analysis of the change process. However, while a wider generality is claimed in principle, in practise a number of issues are not addressed at all. One of the main one is how changes detected at the individual level are propagated and handled at the community level; while this is presented as a precise choice, declaring it out of scope in the foreword,it appears as a major limitation for a tool claiming to support the whole category lifecycle. Also it is mentioned that the model apply only to hierarchical conceptual models, which could be ok but the authors should at least briefly discuss which aspects would prevent usage in different configurations. Evaluating effectiveness of the system is difficult, as only the service tier is discussed, as the UI for the Category Modeller and the Change Recognition are mostly described in term of their implementation, and the Visualiser is still under development. A journal version should surely address the mentioned problems in depth, and even a conference version should also benefit.

Review #3
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] reject
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
-3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
2 poor
== 1 very poor

Review

In this paper, authors motivate the problem of categories that change over time, especially in the context of scientific research. In order to solve it they present AdvoCate, a tool that seeks a representation of this dynamic nature of categories, and records their evolution.

Authors address a very interesting problem, and its importance is well motivated from various angles, including philosophy, knowledge representation, databases and the Semantic Web. They present a novel approach, since some assumptions of the state-of-the-art (e.g. the fundamental facets of categories) are interpreted in an unseen way. The implementation of a system that leverages such assumptions also speaks in favour of the paper.

I have, however, important concerns about the clarity and scope of the paper. Authors claim that there is a lack of ‘connection’ between current ontology evolution tools and processes in science, being current approaches only top-down based, although there exists good data-driven work. The research questions addressed remain, thus, unclarified. Section 2 defines what apparently is an alternative framework for studying changing categories, where concepts and categories are considered different entities, the latter being composed of the classic facets of intension, extension and position in the hierarchy. These definitions and decisions are poorly justified (if any), and never formalised, which confuses the reader.

In Section 3 the approach is described, although not in an interesting way from the point of view of knowledge representation and management. Section 3.1 is too verbose about implementation details, while the rest never goes deep into the techniques and methods followed (e.g. the ‘change identification rules’ seem interesting, but no further detail on how this works is given; no explanation is given on the role of classifiers and machine learning in the pipeline). Authors do not compare their change operations with those of [17], defining their own in Table 1. Concise and detailed descriptions of these issues are needed. A pointer to the implementation source code or demo would be appreciated.

In summary, the paper tackles a fundamental topic for the conference, but it severely lacks concision, detail and formalism, so I suggest to reject it.