Beyond efficiency: A systematic classification of 48 RDFS-based Semantic Web reasoners and applications

Tracking #: 2636-3850

Authors: 
Simona Colucci
Francesco Maria Donini
Eugenio Di Sciascio

Responsible editor: 
Guilin Qi

Submission type: 
Survey Article
Abstract: 
In this paper, we present a systematic classification of 48 RDFS-based Semantic Web reasoners and applications, with the aim of evaluating their deductive capabilities. In fact, not all such applications show the same reasoning behavior w.r.t. the RDF data they use as information source and the ability of reasoning is not a binary quality: it can, e.g., consider or not blank nodes denotation, include different subsets of RDFS rules, provide or not explanation facilities. For classification purpose, we propose a maturity model made up of three orthogonal dimensions for the evaluation of reasoners and applications: blank nodes, deductive capabilities, and explanation of the results. For each dimension, we set up a progression from absence to full compliance. Each RDFS-based Semantic Web reasoner/application is then classified in each dimension, based on both its documentation and published articles. In our evaluation, we did not consider efficiency on purpose, since efficiency could be compared only for systems providing an equal service in every of the above dimensions. Our classification can be used by Semantic Web developers, for choosing a suitable SW system, or to decide at what level an in-sourced application could be implemented and documented, in scenarios in which the three dimensions above are crucial.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject (Two Strikes)

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 04/Jan/2021
Suggestion:
Accept
Review Comment:

All my issues had addressed in this revised version. I have no more new problem.

Review #2
By Bassem Makni submitted on 31/Jan/2021
Suggestion:
Reject
Review Comment:

This manuscript was submitted as 'Survey Article' and should be reviewed along the following dimensions: (1) Suitability as introductory text, targeted at researchers, PhD students, or practitioners, to get started on the covered topic. (2) How comprehensive and how balanced is the presentation and coverage. (3) Readability and clarity of the presentation. (4) Importance of the covered material to the broader Semantic Web community.

First, I would like to thank the authors for submitting their reviewed manuscript.
Even though the authors updated the tables in the document to fix the misclassifications I pointed out in my review, they did not address my concern about their methodology that led to these misclassifications. This makes me less confident that their methodology of relying solely on the documentation is sound. The authors mention a conservative approach for classification i.e when a system documentation does not mention a capability they assume that it does not support it. However, in the case of Jena, the documentation clearly mentioned the derivation support and how to toggle it on. My guess was that the authors looked for "explanation" or "justification" but not "derivation". If this was the case, it would be helpful to provide the details of the documentation assessment and the used terminology for each dimension (and confirm that the updated terminology was used to reassess the other systems' documentations).

I agree with the comment of Reviewer 1 that "When evaluating the deductive capability of the system, you should select some data and queries actually to run the test system and then measure the deductive capacity of the system by comparing the results of the queries."
This is also the methodology I suggested which would have been very valuable for the community especially if a fourth dimension of usability were to be added to the assessment.
However the answer of the authors about the limited time to run such an experiment is not sufficient especially that the approach of relying solely on the documentation was not convincing.

Reviewer 3's comment about explainability was specific to RDFS explainability and not questioning the need for explainability in general. However the authors' comment was about explainability in general.

Minor comments:
"a RDF ..." -> "an RDF"
"to the best of our fairness" -> "In all fairness" or "to the best of our knowledge"

First, I would like to thank the authors for submitting their reviewed manuscript.
Even though the authors updated the tables in the document to fix the misclassifications I pointed out in my review, they did not address my concern about their methodology that led to these misclassifications. This makes me less confident that their methodology of relying solely on the documentation is sound. The authors mention a conservative approach for classification i.e when a system documentation does not mention a capability they assume that it does not support it. However, in the case of Jena, the documentation clearly mentioned the derivation support and how to toggle it on. My guess was that the authors looked for "explanation" or "justification" but not "derivation". If this was the case, it would be helpful to provide the details of the documentation assessment and the used terminology for each dimension (and confirm that the updated terminology was used to reassess the other systems' documentations).

I agree with the comment of Reviewer 1 that "When evaluating the deductive capability of the system, you should select some data and queries actually to run the test system and then measure the deductive capacity of the system by comparing the results of the queries."
This is also the methodology I suggested which would have been very valuable for the communities especially if a fourth dimension for usability were to be added to the assessment.
However the answer of the authors about the limited time to run such an experiment is not sufficient especially that the approach of relying solely on the documentation was not convincing.

Reviewer 3's comment about explainability was specific to RDFS explainability and not questioning the need for explainability in general. However the authors' comment was about explainability in general.

Minor comments:
"a RDF ..." -> "an RDF"
"to the best of our fairness" -> "In all fairness" or "to the best of our knowledge"

Review #3
Anonymous submitted on 18/Feb/2021
Suggestion:
Minor Revision
Review Comment:

We thank the authors for taking all the issues pointed out into consideration, such as making a more precise conclusion, describing other works surveying RDFS-based systems, and discussing the choice of the systems and dimensions.

After reading the reviewed manuscript several times, I have the following extra comments:

1. I do not think the dimensions of their proposed model are strictly orthogonal. The three dimensions, i.e., blank nodes, deductive capabilities, and explanation of the results, are inter-playing. In particular, if a system does not support reasoning, it will not contain the ability to explanation.

2. The authors start both the Abstract and Introduction by introducing their work directly. The Abstract starts with "In this paper, we present a systematic classification of 48 RDFS-based Semantic Web reasoners and applications, ..." and the Introduction starts with "In this paper, we perform a systematic classification of 48 RDFS-based Semantic Web (SW) reasoners and applications".

Without giving the motivations and background of this work, it is hard for readers to understand the significance of this work directly.

3. Why the model is called the "maturity model"? A mature RDFS system does not necessarily support all the features considered here.

4. The sentence "This yielded 48 systems to classify, which were already enough for the comparative tables of a single paper" is strange.

You should give the supporting materials or criteria that make you think that 48 systems are enough. Consider rephrasing this sentence or even dropping this sentence.

5. The newly added two paragraphs, i.e., the two paragraphs of Section 1 before the last one, are also very important but I cannot fully agree.

Our motivation is to let them to make more explanation on why the considered three dimensions are crucial rather than letting them compare these dimensions with efficiency. Besides, the claim "such dimensions are orthogonal to efficiency" is not really correct, since reasoning or deduction effects efficiency. Obviously, query answering by taking reasoning, no matter backward or forward, is less efficient than query answering without considering reasoning.

6. For the "Blank nodes" dimension, why "Consider full denotations for blank nodes" is not included. The authors should make some explanation.

7. In Tables 1 and 2, the rows containing "No mention" are hard to understand. Take the row about FRED as an example. Does "No mention" denote how to process blank nodes is no mentioned or whether it discards triples with blank nodes are not mentioned? What is the meaning of 'x'?