Applying possibilistic axiom scoring to instance-guided ontology evolution in RDF streams

In the context of ontology learning, enrichment, and validation, axiom scoring is the task of evaluating the acceptability of a (candidate) axiom against the known facts. The authors of this article study axiom scoring in a scenario of ontology evolution, whereby new facts (represented by RDF triples) are added to a knowledge base at different times and the underlying ontology needs to be revised based on the newly acquired knowledge.

To this aim, the authors adapt a possibilistic axiom-scoring heuristic proposed in the literature to the case of ontology evolution based on RDF data streams. After defining the problem of axiom testing against a sliding window of a stream of RDF data, focusing on the property axioms of functionality, inverse functionality, transitivity, irreflexiveness, symmetry and asymmetry, they introduce their adaptation of the possibilistic scoring approach and validate it through two experiments.
In the first experiments, they use the CMT ontology to test the extent to which the possibilistic approach is capable of correctly scoring some axioms that are known to hold, using three different sliding window sizes, and compare it to traditional information-retrieval measures like precision and F1-score. The results allow them to conclude that the possibilistic approach is robust and applicable when scoring axioms in streams and with limited data, and more so than a strictly probability-based one.
The second experiment considers an actual scenario of ontology evolution, using the game-related fictional domain of Pokémons, where successive generations (I through IX) provide sets of instances with different properties. To deal with this scenario, they compare three approaches: (i) using the plain possibilistic score, (ii) using same score together with a user-defined threshold for axiom acceptance, and (iii) defining an evolving possibilistic score as a weighted average of the past and present score for each axiom. The results suggest that approach (ii) is the most effective of the three to capture with the ontology the changes occurring in the stream of instances.

The idea of applying the possibilistic axiom scoring heuristics to RDF data streams for ontology evolution is novel and the proposed adaptation of the approach is original.
The article is well-written and easy to read. I found a few typos, which are detailed below.
The empirical validation is convincing, although choosing a real-world ontology of practical relevance would have made Experiment II even more compelling; nevertheless, I am inclined to believe that the domain of Pokémon can serve as a simplified model of the phenomena one could observe in real-world scenarios.
Overall, the paper is technically sound. My comments on the adaptation of the possibilistic approach, detailed below, have to do with the presentation more than with the substantial correctness of the proposed approach.

Detailed Comments

In Section 2 the authors argue that the possibilistic axiom scoring approach of [8] is "implicitly working under the implication of the knowledge available being complete". That's debatable, given that that framework provides for the existence of "basic statements" that are entailed by the axiom being tested but are neither confirmations or counterexamples; in addition, it makes an explicit open-world assumption. On the other hand, it appears that the assumption of completeness is made by the authors in this work, when they state that E0 = E+ + E- (cf. the definition of computeAxiom in §3.1).

In Section 3.2, the formulas for necessity and possibility (conjunctive and disjunctive) can be derived from the formulas proposed in [8] only under the assumption that E0 = E+ + E-. That's perfectly fine, as long as this assumption is justified and made explicit. However, in that case, the formulas of "conjunctive" necessity and "disjunctive" possibility could (and should) be simplified, because when E- = 0, E-/E0 = 0, and when E+ = 0, E+/E0 = 0, yielding:
- for the conjunctive necessity, N(Ax(P)) = 1, if E- = 0, and 0, if E- > 0;
- for the disjunctive possibility, \Pi(Ax(P)) = 0, if E+ = 0, and 1, if E+ > 0.

I find that the justification for using the "disjunctive" definition of possibility and necessity for transitiveness and symmetry opens some opportunities for a more in-depth discussion. As a matter of fact, the two definitions are proposed in [8] to deal with two alternative forms that the logical development of an axiom could take: if the development is in conjunctive normal form (i.e., a conjunction of many "basic statements"), then the "conjunctive" definition should be used; if the development is in disjunctive normal form (i.e., a disjunction of many "basic statements"), then the "disjunctive" definition should be used instead. However, as it is clear from Table 1, here the logical developments for all the axioms dealt with are in conjunctive normal form, because that's what one obtains from grounding a universally quantified formula. This means that the choice of using the "disjunctive" definition of possibility and necessity is not justified by the theory (on the contrary, it would be wrong from the theoretical point of view), but by empirical considerations alone! I think that this should be pointed out, at the very least, and the reasons why the wrong choice according to the theory turns out to work better than the theoretically sound choice in those two specific cases should be further elucidated.

Typos and minor issues:

In the Abstract:
- This paper presented -> presents
- a knowledge evolving scenario -> an evolving knowledge scenario
In the body of the paper:
- different sliding windows sizes -> different sliding window sizes
- references to other sections should be given as "Section 2", etc., and not as "(2)", etc., which is much less clear and potentially ambiguous.
- "possibility theory" does not take the article, much like "probability theory"!
- (Definition 3): one can use to describe -> one can use it to describe
- not considered either a confirmation or counterexample -> neither considered a confirmation nor a counterexample
- Any two individuals with different URIs must be \ considered as / distinct
- Shouldn't #iUri and #pUri read $iUri and $pUri, like in Code snippet 1?
- The experiments described below were done -> ... were carried out
- In code snippets 2 and 3, the FILTER clauses must be wrong - they appear to have been inverted: I think one should read FILTER ( ?o1 = ?o2 ) in code snippet 2 and FILTER ( ?o1 != ?o2 ) in code snipped 3, and not the other way around.
- In Table 7 (and in the text citing it) the negation symbols is not the usual one...
- known to not be present -> known not to be present
- accessing how the relevance -> assessing ...
- The same conclusion can be withdrawn -> ... drawn
- for axioms \ for / which there may be
- An w_p -> A w_p
- With of the introduction -> With the introduction
- in the property was not present -> if ...
- the decision reached \ using / ARI, ARI + cf, ...
- by favouring previous knowledge, it may -> ... ARI_e may
- conjunctive-form using properties -> properties using the conjunctive form

Assessment of the data file

A link is provided by the authors to the GitHub repository of the TICO_Lite tool.
Upon inspection, this repository appears to be well organized, but a README file is missing. I warmly recommend that the authors add one to the repository.
The provided resources appear to be complete for replication of experiments.
All the data artifacts used in the article are there and they appear to be complete.

The submission proposes an approach for the evolution of ontologies in RDF, when the objects are provided as streams.

While the problem studied is interesting, the approach has strong limitations and the formalisation is incorrect. Hence I suggest rejection. Below are some details.

First, the idea of the streams in this case requires that each time a new individual is introduced (or used in the stream) is added with all its properties simultaneously. This is very limiting from a streaming point of view, but is also later ignored by the authors when they speak about the sliding window view, as the same individual may be re-introduced (with different properties) later on, and the approach does not account for this.

The formalisation is wrong in many aspects, but mainly the definition of a confirmation and a counterexample does not make sense, as it refers to one individual only, but properties require many individuals. Moreover, a confirmation only requires that some instantiation satisfies the property. So, for example, the triples (a,r,b), (b,r,a) count as a confirmation to symmetry, even if there are also one million triples (a,r,ci) without the mirror triple (ci,r,a). And the difference for counterexample is between consistency and inconsistency, which again is an inadequate measure.

The work needs a much better formalisation before it can be accepted.

== Abstract

The authors describe a stream setting in which (RDF) facts are presented and the task is to evaluate the nature (called axioms) of the attached properties. The paper mainly consists of two case studies, evaluating the approach. The scenario of evolution is simulated by the use of data streams.

== Evaluation

While the paper is framed in terms of OWL and RDF, and the data sets in the experiments might reflect these formats as well, the actual methodology seems to be independent of these choices. The use of a somewhat open world assumption makes it a fit for the Semantic Web community. While the paper is, over all, easy to read and the authors sort it into the realm of logic-based ontology (re-)learning, the main contribution is neglecting the logical aspect. The considerations are somewhat narrow to what the authors call the possibilistic approach. When I see learning of logical rules, I also expect some consideration of association rules, which also deal with open vs. closed world as well as incompleteness in their data. How does your approach actually related to ILP and those association rule approaches? Would a partial completeness assumption imply a change in your own methodology?

When the authors define their problem (Sect. 3), the reader is overwhelmed with an (in-)formal wall of definitions and no illustration of their concepts. It is rather informal since I can see a TBox and an ABox, and certain familiar DL expressions, but no semantics is given. The semantics seems to be not important, but so is the introduction of what an ontology is, for instance. For the axioms of interest, the authors build on an intuitive understand of first-order logic anyway. But then in Def. 4, a certain understanding of the modelhood ($models$) is required. Regarding Def. 4, the authors state that examples of $Ax(P)$ are identified by substitutions admitting the implication, but then scenarios not adhering to the body are not counted. Isn't that contradictory? Another occurrence where formal understanding of ontologies and their semantics seems to be required is Def. 5, where the undefined notion of consistency is assumed to be known. For a potential revision, please think about the notions required for the reader to actually understand what is happening in your work. Counting certain patterns, as the result of translating first-order style axioms, certainly does not need the full (or even half of the) machinery of OWL/ontologies.

When it comes to functions like evolve, I find the functional notation quite unusual: it seems the arrow $->$ is something like the equality symbol. Also, $Ax(P)$ gets at least two different formal meanings: first, it is an element of the axiom set, later on abbreviated; second, it stands for the first-order representation of the axioms. The notion of support is multiply defined, first by evaluating the axiom formulae and then by the counter $E^+$, $E^-$, and their sum, apparently called $E^0$. How do these notions relate exactly? Are they the same? Please clarify.

Regarding now the main scoring technique, the ARI, I found it hard to evaluate the different approaches. First, there is the formula introducing two unknown notions, which are then discussed afterwards. I would switch this introduction. Especially here, it is not clear if the window or the whole set of items is considered. Here, I would expect a far deeper relation to the timeframe and/or the whole setting of streams, which is part of the title anyway. Also, the formulae are just given without further motivation; why squarerooting, why quadraticly increase? Is that state-of-the-art? As a minor remark, in the second set of formulae for $N$ and $Pi$, why is the $E^0$ not subscripted with the property axiom in question? After all, if we take the streaming scenario seriously, I would expect to take some trend position for the evaluation.

Overall, I can see the evaluation to be published, but I would need a major revision of the problem definition and the actual methodology. The authors refer to another paper, but a bit more self-containedness is needed. I especially need to be able to relate all the technical terms in the title to the paper's content and methodology. How are the data streams, i.e. the sliding windows, exactly utilized? The method I read in the paper introduces some notions regarding the streams, but the actual scores are agnostic of the scenario, making it hard for me to evaluate the actual contribution of the paper.

== Minor Remarks and Suggestions

- Table 1 is somewhat misplaced; the text columns above and below are to be read together, which is unusual. If table 1 was at the top or bottom of the page, there is no confusions to be expected.
- there are some typos here and there; please check the language for a potential revision.
- instead of the wall of definitions, a descriptive text, merged with the examples coming later in the current paper would be much appreciated by this reviewer
- wording-wise I would expect the notions to Reflexivity, Transitivity, Irreflexivity, and so on. But if other reviewers are fine with the notions presented, I will not insist.