loomp - mashup authoring and semantic annotation using linked data

Tracking #: 1472-2684

Authors: 
Annika Hinze
Markus Luczak-Rosch
Ralf Heese
Hannes Muehleisen
Adrian Paschke

Responsible editor: 
Ruben Verborgh

Submission type: 
Full Paper
Abstract: 
Legacy full-text corpora and low precision of automatic annotation have long hindered widespread utilization of semantically-rich content. Best annotation quality on specialised collections can be achieved through manual annotation; most manual approaches assume that semantic annotations will be created by domain experts. However, the poor usability of many tools is a challenge to end-users who do not have extended knowledge of semantic techniques. For the semantic content authoring tool loomp, we followed a user-centred design approach to implement mashup and manual annotation of textual resources. Loomp was developed in close collaboration with knowledge workers and domain experts from public service institutions. This article provides technical details of Loomp's design, architecture and data model based on requirements derived from real-world journalism use cases.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Simon Hengchen submitted on 26/Oct/2016
Suggestion:
Minor Revision
Review Comment:

This paper presents loomp, a semantic web-based annotation tools for non-expert end users. After presenting a thorough review of the literature, the authors raise the user requirements for such a tool, and describe them comprehensively. The technical details are then produced, along with screenshots and how the user requirements are implemented.

Whilst the research is not exactly new -- as per the authors, the paper describes the technical details of an already presented tool --, the submission brings new information to the table, and by doing so enables researchers and practitioners alike to use loomp, the tool described.

The evaluation of such a tool is difficult, but the authors provide an extremely well documented description of other similar work which helps in doing so. Nonetheless, comparing similar tools on criteria/requirements around which one's tool was specifically made does not bring much to the table.

Despite a few typos and the shifts from British English to American English (-ise vs -ize, for example), the paper is well written.

The major flaw of this paper regards the evaluation process of the entity selection by the end users (Section 3.4, §2). The authors mention that "all participants were able to select the correct concept from the list if the entries contained enough information to identify the entity they had in mind". The paper presents no information on :
1/ What the GSC is -- how did THEY determine what the correct concept was? Was it done by one person? Several? If yes, was an inter-annotator agreement used? (Cohen's Kappa, for example)
2/ What is considered "enough information"?

Should those points be clarified and the minor typos* be addressed, the submission should be accepted.

Review #2
By Tim Clark submitted on 01/Nov/2016
Suggestion:
Accept
Review Comment:

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing.

This paper describes loomp, a mashup authoring and semantic annotation tool developed to journalism use cases. While it is not in itself original to attempt development of semantic annotation tools, in this particular case the well-documented and thoughtful methodology used in develping the tool is highly instructive. In particular the key observation that annotation in context needs to be a byproduct of useful activity for journalists and not a thing in itself, drove a number of characteristics of the tool. So the results are significant in terms of applying proper methodologies, requirements analysis, and thiking through the problem.

This reviewer did not download or attempt to install the loomp package. Whether loomp is long-term a useful tool is an open question and is the proof of whether requirements analysis of the authors reflected real needs and their product met these needs. I note only that github shows a last commit of actual code in February 2015, so perhaps deployment to a user community was not in the game plan. It is always an interesting exercise to balance interesting research topics in computer science, with the practical needs of ones users.

The quality of writing is excellent with completely perfect English usage and transparent, easy-to-follow exposition of the topics.

I see no need for revisions. This article should be accepted.

Review #3
By Antoine Zimmermann submitted on 22/Jan/2017
Suggestion:
Major Revision
Review Comment:

The paper describes the system loomp, a software program for authoring, annotating, and mashing up textual content. The tool is designed in such a way that it minimises the overhead of annotating text, as well as the degree of expertise in semantic technologies required to add annotations. An important part of the design principles relies on Linked Data principles and techniques. The main evaluation method is by way of listing fulfilled requirements and comparing this to how other existing tools fulfil them. Results seem to suggest that loomp addresses a larger number of requirements than currently available competing tools.

The paper is quite well written and clear, making it a nice technical documentation for loomp. Loomp itslef utilises interesting design choices that are apparently relevant. However, I believe that there is an important flaw in the way the contribution is evaluated. If the contribution is a design approach which better addresses the requirements of authors-annotators-mashup creators, then I deny that the paper is convincingly showing that. If the contribution is that such design approaches make it easier to use an annotation tool such as loomp (or faster, more efficient, etc) then I don't think the paper is proving that.

The main evaluation technique that the paper is relying on is by comparing requirement fulfilment by various tools in the same category as loomp. Unfortunately, this would only work if the requirements had been evaluated themselves in the first place. If the requirement analysis identifies a requirement that is in fact of minor importance, while it leaves aside an important requirement, it may unbalance significantly the importance of the authors' achievement. The way the requirement analysis has been done, according to the paper, is fair, but there should be an extra step for assessing if this analysis is correct and complete.

A set of requirements that are properly validated would be a nice contribution because it could be reused and serve as a benchmark for the evaluation of other tools in the same category. However, such validation may be very difficult. Another approach is to ignore the evaluation of the requirements, using them simply to guide implementation, and evaluate the final tool in a different way, e.g., with a user study. The authors mention an existing user study in 3.4 but it's very succinct. A user study could, for instance, reveal that the sheer number of requirements addressed is irrelevant and that a tool that addresses a few key requirements very well is much preferred. With the current version of the paper, it is impossible to say whether loomp is an actual improvement over other tools, beyond possibly being personally convinced by the intuition.

Consequently, I'd recommend that the authors prepare a new version of the paper where they concentrate on improving the evaluation, either with better justification for the set of requirements, or by more detailed user study, or at least by discussing the pros and cons, especially the limitations, of the current evaluation methodology. This may lead to changing and restructing the main arguments and I think it's a major revision to be made. Another option could be to submit the paper to the "system description track" of the journal.

I have a number of extra comments that justify more work on the paper too.

Detailed comments:

- There are inconsistencies in the way some words are spelt: sometimes it's in American English, sometimes it's in British English, even for the same word. E.g., in the abstrace "utilization" vs "specialised" in two consecutive sentences.

- Several times, you have multiple references written, e.g., [3][6][7][8][12] while sometimes they are written [3,6-8,12]. Prefer the latter.

- Sec.2.1: "Annotation is not regarded the main task" -> regarded as the main task

- Sec.3.2:
* "Frankfurt(Oder)" -> missing space (occurs twice)
* "marked green" ... "in red" -> while I understand this refers to the colours in the interface of the tool (which you want to be colourful), the text should be clearer what this is referring to for people who are reading the article in black and white (as I do on printed pages).
* "the authoring window (C)" -> I think you mean (A)

- Sec.3.4: "w.r.t novice" -> missing dot after 't'

- Sec.4.1:
* there should be a proper bibliographic reference to RDFa instead of a mere URL in footnote:
Ben Adida, Mark Birbeck, Shane McCarron, Ivan Herman. RDFa Core 1.1 - Third Edition, Syntax and processing rules for embedding RDF through attributes, W3C Recommendation 17 March 2015. http://www.w3.org/TR/2015/REC-rdfa-core-20150317/
* "html and RDFa" -> HTML and RDFa

- Sec.4.2.1:
* Fig.4 is using colours to differenciate kinds of entities which I can't distinguish in B&W.
* In Fig.4, I don't understand why the terms in a vocabulary are in sequence. What does it mean that a term is the first of a vocabulary?
* I don't see the rationale behind the design choices of the ontology in Fig.4. Why "NamedEntity" isn"t simply a subclass of a class "Entity"? Perhaps even, a subclass of owl:Thing (if ou want to stay in OWL DL), or of rdfs:Resource? Perhaps you need "annotationDomain" to relate to a class? then why not use "rdfs:Class" instead of "EntityClass". Do you want/need it to be an OWL DL ontology? Besides, the figure looks like a UML diagram. How are we supposed to interpret it in terms of RDF or OWL?
* Here, "mashup of" seems to imply "sequence of". Is it really the case, from an ontological point of view. I mean, of course you can use a sequence, or whatever implementation of collection you like in your tool, but the ontology doesn't have to mirror your implementation.
* "external ontologies, such as RDF or FOAF" -> RDF is not an ontology
* "rdf:label" -> rdfs:label
* "is encoded as the property rdf:label [sic] with the domain http://dbpedia.org/resource/City" -> do you mean something like "rdfs:label rdfs:domain dbr:City" ? I hope not, because it's very very bad practice to redefine or restrict the domain and range of a standard property.
* "a vocabulary therefore consists of a list of terms" -> why list? see comment above on "sequence". Especially, there should not be a "therefore" because this does not follow from what precedes.
* In Fig.6 "annotationProperty" is in a rectangle as if it were a literal. Isn't it supposed to be the URI owl:AnnotationProperty? Why would it be written in CamelCase if it's merely a literal. Also, the other literals do no have language tags. Is it on purpose?
* "For clarity, we refrain from showing a complete serialisation of the ontology" -> how is it clearer to not have the actual definition of the ontology? If the paper was written for scientific popularisation, it would of course be clearer to a mondane reader to have only plain English explanations. However, it is supposed to be a scientific paper for the scientific community. Ther eare other reasons why you could omit the complete serialisation, e.g., that it is too long or that it does not bring much. But I think that having the actual ontology in appendix (if not overly long) or in a persistent web page with a link to it in the paper, would be clearer.

- Fig.5:
* loomp:Fragment "each loomp Fragment is an instance of loomp:Fragments" -> this description is not really useful. Anything that says "ex:X - each x is an instance of ex:X" is uselessly redundent. Moreoever, it's loomp:Fragment, not loomp:Fragments.
* loomp:AnnotationSet is described as a sequence rather than a set. The name is not well chosen. Also, why this is a subclass of rdf:Seq, rather than rdf:List, or anything else? It seems that this is describing an implementation choice for the application loomp rather than a domain ontology of documents, document mashups and annotations. An application that deals with annotation sets purely as sets is dealing with the same ontological concept. It would make more sense to define AnnotationSet as a bunch of terms (a class that has rdfs:members that are Annotations) and possibly define a subclass that's internal to loomp where annotation terms are ordered. The fact that the terms are ordered is an implementation choice of loomp that has nothing to do with the ontology of document mashups.
* loomp:annotationDomain "relates a term in a loomp vocabulary to a class of named entities" -> why "named" entities? why not a class in general. The classes in DBpedia are not necessarily classes of named entities. Considering the way DBpedia classes are used all over the Web, it is clear that it is not always interpreted as a class of *named* entities.
* loomp:hasRDFa and loomp:contains -> the description is the same. Moreover, it's rdfs:Literal, not rdf:Literal.

- Sec.4.2.2: "The documents' full texts" -> The full texts of the documents (two occurrences)

- Fig.7:
* there are two types of dashed lines, on that is thick, one thin. What's the difference?
* "Frankfurt(Oder)" -> should be "Frankfurt (Oder)" (two occurrences)

- Fig.8:
* why is it useful?
* it is truncated on the right hand side
* the text in Sec.4.2.2 says that it is the raw RDFa data. It's actually RDF/XML

- Fig.9: the loomp:type is a character string. Is it on purpose?

- Sec.5:
* """titled "Transport Points".""" -> actually, it's titled "Transit Points"
* "Figure 5, the user is now" -> In Figure 5, ...

- Sec.6: "in Table 17" -> it's Fig.17

- Sec.6.1.1:
* "S-CREAM is framework" -> is a framework
* "supports the capturing and semantically annotations" -> and semantic annotation
* "through deep linking" -> what's this?

- Fig.17:
* "Interoperabilty" -> Interoperability
* "vocaulary" -> vocabulary
* what's "supported" as opposed to "fully supported"? Should'nt it be "partially supported" vs. "fully supported"

- Sec.6.1.2: "has the potential become" -> to become

- Sec.7: "PROV [57] is not yet widely adopted" -> is it? proof?

- Acknowledgement: There are straight double quote symbols where there should be closing quote symbols

- References:
* There are acronyms that have been lowercased. Use '{' and '}' in BibTeX to ensure capital letters are preserved.
* ref.5: "–"
* ref.15, 17, 19, 45, 52 have an extra dot at the end of the titled
* ref.32 does not have a date of publication

Review #4
By Maria Maleshkova submitted on 26/Jan/2017
Suggestion:
Major Revision
Review Comment:

This paper presents an annotation framework that aims to support the annotation of content by non-experts.

General comments:
1. A general recommendation is to be more careful when making statements and precise when introducing terms. Currently, there are a lot of vague statement or statements that are not backed up by evidence. A couple of examples:
- “Legacy full-text corpora and low precision of automatic annotation have long hindered widespread utilization of semantically-rich content.” The precision really depends on the domain and the type of text. Some annotations, entity recognition approaches, etc. have quite a nice precision in the mean time. Has the low precision of annotation hindered to utilization or semantic content? We are seeing more and more applications that use semantics…
- What is a mashup? You need to give a definition early on because the terms has already a couple of meanings, for example an application based on API compositions
- “The legacy of existing document archives and other fulltext corpora without semantic annotations is one of the major hurdles for a broader acceptance and availability of semantic techniques,” Semantic technologies are witnessing growing acceptance exactly because of the potential for data-integration solutions. Furthermore, there is a lot of new content generation (Sensor data, social sem. Web), which is completely orthogonal to legacy data. Where do you find evidence for this statement
- “However, most of these tools follow indiscriminate brute-force methods that
lead to low-precision markup [9].” Or maybe the problem is not that the methods are ‘bag’ but rather that the task is simply very difficult.
- “Semantic enrichment of those new documents should be integrated
into the authoring process.” Why? There is no evidence that this is the best approach.
2. The introduction very much focuses on why other approaches have shortcomings or are inadequate. Instead, give a very clear problem statement and specify exactly what challenges in the context of annotation you want to address. List the contributions that you make. Discussing the advantages and disadvantages of other tools and comparing them to your work should be done in the related work section. Overall, it is not good research practice to start a paper by saying the everyone until now has done poorly.
3. How specific is the tool to the descried scenario? You need to clearly state if there is a general applcicability or it is rather a custom solution. I would suggest having scenario-specific design requirements and extending these with general requirements, in order to show that the work have a general impact.
4. Step 2, p2. What is a text fragment? I would suggest adding a definition. Is a text fragment the same as an article?
5. Step 3, p3. “based on expert semantic markup” ? But the annotations are actually personal annotations???
6. Section 2.2. Add an explanation of the method for deriving requirements. Guided interviews with the journalists? Only observation??? How was this done.
7. R1 is a bit surprising. This is the first time a mashup is mentioned (very prominent in the title but not in the introductions or the scenario). What is a mashup?
8. R7 what about the language of the articles? Does this have an impact?
9. R12 – why is this context? Usually, something like users in the same location would get similar annotation suggestions. Define how you use contextual semantic in the scope of you work.
10. “Once the fragments have been assembled within the mashup, their origin
(new or re-used) can no longer be deduced from the interface.” Is this good? What about R8?
11. Where are the tools and the demo from Fig2. available?
12. Section 3.2 What happens if multiple snippets have the same annotation?
13. Section 4.1 How are existing resources outside the system accessed and linked ?
14. 4.2 How was the ontology designed? Design patterns? Ontology engineering principles? Is there a paper where the ontology is already published and described in more detail?
15. RDF does not have a Lable, RDFS does. “Frankfurt (Oder) is encoded
as the property rdf:label with the domain
http://dbpedia.org/resource/City. “There are supposed to be real examples…

Detailed comments:
-- “Throughout the project” p2. Which project???
-- Step 1, p2. Targeted search can be facilitated in may other ways… rephrase.
-- “and then a concept from the offered vocabulary is selected” can you be more specific? Which vocabulary are you actually using? Would the tool work for a medical use case?
-- Fig 4. The labels of the properties are messed up.


Comments