Aethalides: An Ontology-Driven News Classification Framework

Tracking #: 1540-2752

Authors: 
Wouter Rijvordt
Frederik Hogenboom
Flavius Frasincar

Responsible editor: 
Guest Editors ML4KBG 2016

Submission type: 
Full Paper
Abstract: 
The ever-increasing amount of Web information offered to news readers (e.g., news analysts) stimulates the need for news selection, so that informed decisions can be made with up-to-date knowledge. Hermes is an ontology-based framework for building news personalization services. It uses an ontology crafted from available news sources, allowing users to select and filter interesting concepts from a domain ontology. The Aethalides framework enhances the Hermes framework by enabling news classification through lexicographic and semantic properties. For this, Aethalides applies word sense disambiguation and ontology learning methods to news items. When tested on a set of news items on finance and politics, the Aethalides implementation yields a precision and recall of 74.4% and 49.4%, respectively, yielding an F0.5-measure of 67.6% when valuing precision more than recall.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 28/Feb/2017
Suggestion:
Reject
Review Comment:

The submission presents the Aethalides framework which performs an ontology-based news classification by applying word sense disambiguation and ontology learning methods to news items. The framework has been evaluated on a state-of-the-art corpus.

(1) originality
The submission is insufficiently original. The work combines a set of state-of-the-art solutions to address a not novel problem. The related work section is a plain analysis of other researches in the fields of news personalization, WSD, ontology learning, but does not highlight differences compared to the proposed solution. A large part of the paper describes techniques, resources, methods, and tasks well known in the Semantic-Web community, and does not provide any significant contribution to the reader

(2) significance of the results
The experimental evaluation is carried out on a quite small dataset, and the majority of the experiments evaluate state-of-the-art techniques used by the framework. This evaluation shows a little significance, since the preliminary assessment that a given technique is suitable for the defined research goal can be reported in the results, but can not represent the main part of the experiment

(3) quality of writing
The paper is well written, easy to follow and understand, but shows a small contribution for advancing the state of the art in the semantic-web research area.

Review #2
Anonymous submitted on 20/Apr/2017
Suggestion:
Major Revision
Review Comment:

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing.

The paper describes the Aethalides system for knowledge-based annotation of news items. The envisaged application area are financial news, but there is no discussion of the specifics of this domain, and the evaluation also shows only limited evidence of domain-specific patterns or problems.

The system is a pipeline consisting of standard components of text analysis, including word-sense disambiguation. All components use well-known techniques or small variants thereof. It is unclear what exactly the claim to novelty/contribution is.

Results from a detailed, component-wise evaluation are reported, but a comparison with competitor systems is lacking.

The system might be a very useful system for real users in a real domain, and this could constitute a substantial contribution to the field. However, no evidence of such an application, or of the collection of user reactions, is given. If there is no such user base, an alternative would be to offer the system as open-source code, so people can use it and build on it.

A substantial number of other papers involving the authors is cited, but it does not become clear what the contribution of *this* paper over the existing publications is.

The writing is in general good (except for some linguistic mistakes, see below) and sometimes a bit long-winded, as when basics such as precision and recall are explained.

In sum, while this looks like sound work, it is unclear what the scientific novelty and contribution are. It is also not clear why this should appear in a Semantic *Web* journal. This omission could probably be fixed easily, by integrating the system with WWW components, e.g. to process online news.

Some minor comments follow.

p.1: transition to usefulness for financial applications a bit abrupt. Is that the only application? (p.2 gives the motivation that your system is for this domain, but the reader doesnt know this yet on p.1)

amount of items -> number of items

weaknesses of SemNews?

How do you determine that KIMO is minimal, and how that it is sufficient (and for what)?

2.2. first sentence is not a sentence.

corner stones -> cornerstones

" [37]. They distinguish: please use the authors names, otherwise they is undefined.

" Extraction of relational hierarchies, axiom schemata, and general axioms is seldomly explored: Isnt, e.g., machine reading (Mitchell et al.) specifically dealing with this?

will only yield acceptable results for common

general relations: what is that? For what relations does the approach not yield acceptable results? How do you know? (Is it in the paper cited at the beginning of the bullet point?)

" is a system of which the creation was prompted by the lack of methods for the extraction of relations between concepts other than hypernyms.: can you simplify this sentence? Also, why do you talk about the motivation of this system, and not of the others? And why this motivation, if there are all these other systems (you described them in the other bullet points) that do extract non-hypernym-relations? (Please also be careful with divergent uses of one term. The sentence would be easier to read if you used hypernym relations when you talk about the relations.

Section 2.3 lists a number of systems, each of which has apparently been the subject of one paper. (Were these systems all one-offs?) But no bigger picture emerges. Also, it is not clear whether all these systems were developed in a disjointed fashion, or whether there was some progress in the area.

It is also unclear what is still missing in the research area, and therefore which gap you are trying to fill with your system.

attached might -> attached to them might

patterns, that -> patterns, which

GATE application consist -> GATE application consists

The similarity measure in formula (1) is unbounded. I believe it needs an extra component to yield a similarity measure between 0 and 1.

serve this propose -> serve this purpose

" Practically, this means a smaller memory footprint and a possibility for parallel processing. : did you test this? What were the results?

What makes the content of Section 4.5 a life cycle? Is WSD dead/invalid/etc. after that? Isnt this simply a WSD process?

I dont understand why SSI should require monosemic words. Wouldnt it just pick out the smallest/most densely connected semantic graph as its interpretation?

Was your domain ontology machine-learned or hand-crafted, or both? If the latter, in what proportions?

golden standard -> gold standard

The component-wise evaluation is very nice, but I dont really understand the contribution of this part: from the text before, it appeared that you were using standard toolkits/APIs for the different processing steps? So does this section constitute a test of existing software with regard to the standard corpora you use?

Are these standard corpora really representative of your real-life input material? If yes, how did you establish this? If not, what does this imply?

How do your evaluation results relate to other state-of-the-art systems?

And what does all this mean for the news consumer?

Review #3
By Giulio Napolitano submitted on 06/Oct/2017
Suggestion:
Reject
Review Comment:

This paper describes and evaluates an extension to an existing system for text categorisation, used for filtering documents for users in the financial domain. The authors are particularly interested in speed and precision performance and state that their focus is mainly on semantic technologies for word sense disambiguation (WSD), a crucial challenge for text classification tasks. In particular, they also state their work belongs to the domain of ontology learning methods.
The system being extended employs a manually generated and curated financial ontology, which is linked to Wordnet synsets for automated (rule-based) augmentation. The proposed, extended system adds a word sense disambiguation component to the original pipeline, which employs the (existing) SSI algorithm.
The paper is written very clearly, although some corrections and polishing are still needed, as well as some minor clarifications.
My first concern, however, is about the substance of this contribution, both in terms of originality and extent. The Compounder and the WSD component are, as far as I can see, the only additions to an existing system, with the WSD component itself using an implementation of an existing algorithm for the task.
I’m also not sure about the stated focus on ontology learning. What is here proposed on this topic? Apart from refinements of some accessory components, the real contribution does not seem to me in the ontology learning domain, unless the authors actually mean “ontology population”? Isn’t the same Hermes ontology used here?
Assuming that the central contribution is in an extension to the Hermes framework, the detailed evaluation of the new framework misses a comparison with the old one. How exactly can the authors conclude this extension represents an improvement?
At a lower level, I’m also not convinced the formulation of the recall measure is appropriate. Here I would rather include the incorrectly classified in the denominator. In the case of the WSD component, this would make its recall equivalent to its precision but it would still be more sensible than an uniformative 100%.
The detailed reporting of the performance of all the other components, which have not been developed by the authors, might be excessively long. Also, the WSD component performance is compared to other SOA tools, but there is no direct comparison on a common dataset which would be much more convincing.
Finally, some statements (e.g. “When a word is disambiguated, it can be used in further runs of the algorithm and may help disambiguating other words.”, page 7) might be further explained.
In summary, before having this paper reconsidered, the authors should (as a minimum):
1. Clarify what is the focus of their contribution.
2. Make sure the recall measure is properly defined and calculated.
3. Compare the performance of the extended system to the system being extended.
4. Improve some explanations and rebalance the paper sections.
5. Have an independent native speaker of English check again the general correctness.
Less work would be required if the authors decided to submit this work to a suitable conference or workshop, which I believe should be a preferred option.


Comments