Towards Linkset Quality for Complementing SKOS Thesauri

Towards Linkset Quality for Complementing SKOS Thesauri

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

-1
Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)
3

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

3

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
4

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
3

Review

This paper describes a measure for determining the quality of (VoID) linksets that interlink SKOS thesauri. The introduced measure, termed “linkset importing”, determines whether the linkset will increase the the completeness of a selected target dataset. Completeness is determined by whether new values for properties in the target dataset can be incorporated via the linksets. This would be helpful for example in determining whether additional language labels for resources would be included.. The evaluation is done with respect to a synthetic benchmark. The authors describe the benchmark generator and make it available.

Overall, I like the idea behind the paper. Links are a crucial part of what makes Linked Data, linked data. We see that many of the interconnections are generated by third parties in the form of linksets. Indeed, there may many different linksets that could be potentially of interest, thus, understanding whether any given linkset enriches a particular dataset is clearly of interest. (On this point, I’m not sure if the author’s statement that link sets are extremely important in Linked Data is provocative as stated in the conclusion.)

Unfortunately, the paper suffers from two major issues: terminological coherence and breadth of evaluation.

## Terminological coherence

While reading the paper I struggled understanding the definitions of important concepts.

Some examples:

- The definition of linkset importing is given in a couple of different ways: “linkset importing, a measure which assesses linksets as good as they improve a dataset with its interlinked entities’ properties.” It’s unclear what improve means or interlinked properties. Or ““linkset importing which measures the percentage of values that can be imported in subject dataset, from the object dataset, when complementing via a linkset.” It’s not clear what the subject and object dataset are in this context.

- The notion of complementing is central to the paper but it’s presented in a confusing fashion e.g. ”For completeness of dataset obtained by complementing SKOS thesauri with their skos:exactMatch-related information”. What is skos:exactMatch related information? It took me until section 3.2 to figure out what was really meant with the definition when the scoring function presented with respect to the notion of importing potential defined as “this function evaluates how many new values for a property p, are distinct from those already existing in the subject dataset X and can be reachable through L”.

- RDFEntities is oddly defined as entities “exposed as” resources. I assume the authors just mean all the resources within the datasets considered. Sometimes within the literature there is a distinction between resources and entities within an RDF dataset. It would be helpful to be precise.

- In the definition of the test framework, T is defined as the seed thesaurus which acts as the gold standard but later when discussing completeness assessment T is defined as a thesaurus in one the tests sets: “thesauri T, G ∈ D, where T is a subject thesaurus in one of the test sets, and G is the gold standard”

These are just some of the examples. I think the whole paper would be improved with more precise terminological definitions. In particular, I think what the author’s trip over is the idea that one is “adding” a dataset to another one via a linkset. I think by specifically adopting the terminology from VoID, void:subjectsTarget and void:objectsTarget the terminology would be much less confusing.

## Evaluation

In terms of evaluation, given that it is synthetic, I wondered why the evaluation was not tested with more seed ontologies? Likewise, not all results were presented. I would prefer more results in the paper rather than the validation architecture picture, especially, given that the OAEI framework is used a reference architecture.

For Figure 3, why is the plot drawn as a connected line chart given that the x-axis are the test sets considered? It would also be tremendously helpful to have a better connection between Table 2 and Figure 3. Table 2 itself is difficult to understand. It’s fundamental to the paper and difficult to understand what the various changes actually are. For test 1 and test 2 it seems it’s based around the deletion of percentages of skos:labels in the test. I’m not sure what modifications were made for the remainder of the test datasets.

Some final thoughts:

In terms of related work, while the summary of the author’s prior work was fine, what I missed was the differences to this work. It focuses on SKSO, ok, but what is being reused, added upon or changed?

I didn’t understand why the focus was on SKOS, couldn’t the method be applied to any linkset?

I also wondered about the limitation to measuring just values for properties. What happens if the object-target has properties that aren’t in the subject-target? Don’t those additional properties increase the completeness of datasets.

In summary, I think the core of the research is there but in the current setup it is difficult understand and thus the measure itself cannot yet be built upon by others or compared against.

Minor comments

- “datasets has been exposed and interlinked according to…” - has been should be have been
- “proposing the linkset importing” - remove “the”
- Can you provide support for this statement: “Linked Data is largely adopted by data producers such as European Environment Agency, US and some EU Governs, whose first ambition is to share (meta)data making their processes more effective and transparent.”
- Add a citation for this quote “Linked Data will evolve the current web data into a Global Data Space”
- “remarkable number of thesauri” - how many?
- “LACT link specification” should be “LATC link specification”

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
2

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)
4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
5

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
3

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Review
This paper presents a novel quality measure for assessing the quality of sets of links between SKOS concept schemes (thesauri). The main idea is to measure the gain one makes when 'importing' the properties of target concepts onto the concepts of the source concept scheme. The paper presents the bases for the measure, and an evaluation in the context of an artificial benchmark where the behaviour of the measure is tested against different variations in terms of information present in the source and target thesauri and the amount of links. In the light of the results the authors adapt the measure.

The paper has many issues, some presentation ones (glitches in formalization, and general verbosity), and some bigger disappointment in the scope of the work done. After all, this is just about one measure, and there is no evaluation for cases of real (different) concepts schemes and linksets. The evaluation reported is also in the light of just one property, skos:prefLabel. Finally one may doubt, whether the benchmark really captures all the dimensions of the problem space, which is never fully characterised in the paper.

However, many of these issues can be fixed quite easily for the final version. In general the paper is clear, and seems a fair attempt at enhancing the state-of-the-art, worthy of presentation for EKAW. In fact this measure stands a good chance to be actually re-used in efforts that produce or vocabulary alignments. It really seeks to capture the contribution of an alignment, taking into account the data richness of the target concept scheme, and the possible weaknesses of the source, which would need completion. This reviewer as a potential 'industrial' use, appreciates the possible value of this approach.

More detailed comments:

- many use the word 'thesaurus' to refer to SKOS 'concept schemes'. It is not really bad since SKOS is used to represent many thesauri and was inspired by their models. But it can be used to rperesent other knowledge organization systems.

- it can be confusing to refer to "predicates" that after a first binding are still predicates. for example in section 2 the authors introduce the predicate "t", but after that all formulas use its application to D "t_D", which is itself a predicate. It makes the paper easier to read, but still...

- 3 is sometimes verbose and has formalization issue.
The categorisation of quality indicators, scoring functions and aggregate metrics are not extensively re-used later, and I argue they are not crucial for the paper.
The intro of notions could be streamlined (just go to definitions!) and some sentences are awkward anyway ("We present Void-inspired indicator", "Let X be an dataset, p be an RDFProperties, and e be an Entities(X)"!).
The last formula of 3.1 is quite impossible to understand on a closer look, even though it paradoxically seems ok to see what [Z]_L is in fact. It is not a good sign that Z doesn't appear in the right-hand side of the equality. And it can be probably be much simplified.
I assume LinkImpl4p_L is 0 when den is 0, not when it's different from 0.

- RQ1 and RQ2 are really two sides of the same thing, and considering the 'evaluation' that is made of each of them I would recommend to combine them to make it a stronger and clearer point. RQ3 doesn't appear very elaborately tested, either.

- there's a lot of repetition the text of 4.2. In fact the figure 1 is not really needed, as the text is already telling what's happening (which is itself not so complex).

- in several places the authors report having done evaluation with skos:altLabel, but since it is not reported, this could be removed altogether. By the way GEMET concepts have very few, if no altLabels (the English definitions download at http://www.eionet.europa.eu/gemet/exports/en/rdf/ contains none), so it would be really interesting to see the results.

- the formula for the normalized version of the score is not presented, even though the text mentions the coverage coefficient.

Overall evaluation
Select your choice from the options below and write its number below.

-3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

4

Novelty
Select your choice from the options below and write its number below.

2

Technical quality
Select your choice from the options below and write its number below.

3

Evaluation
Select your choice from the options below and write its number below.

2

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

2

Review
In this paper the authors discuss the notion of link sets quality and propose a metric to look at the number of additional predicate/value combination gained from linking one concept to another. The metric is introduced in detail and a very short evaluation is provided.

I have several issues with this work starting with scoping what is exactly assessed and measured. There are references made to "quality of exposed data", "link importing", "linkset completeness", "value of interlinks" and "property completeness" while it is stated that "linksets [are] good as they improve a dataset". The wording could be clearer to state that what is evaluated is not the quality of the underlying data exposed as LD, nor the quality of the conversion work, nor the quality of the ontology alignment. What is measured is just how many additional p/o pairs concepts from a given ontology gained by having links with another. Which is actually the major flaw of the paper: with such a metric, making *wrong* links is the best as you are more likely to gain skos labels that are not part of the initial description !

The evaluation section is hard to decipher and it was thus difficult for me to check this from your results but looking at the formalism introduced a large set of random links between two ontologies should bring better quality results than a smaller high-quality (as evaluated by OAEI indicators) set of links. Enriching the description of an entity can be an indicator for the usefulness/value of the link(s) used, and maybe contribute to evaluating the link quality but as such can not be considered as a single quality metric (see [7]).

Some more details:
* In Section 2 it is surprising to read that skos:exactMatch is considered for importing triples from another source while this relation works both way. A more exact definition should rather express how X and Y enrich each other from the existence of these bi-directional links
* In Table 1, L in not necessarily a part of D whereas RDFProperties should be a subset of RDFEntities
* In Definition 4 it is unclear what "den" is. By the way it is also unclear why 3 pages of complex formalism is needed to introduce that the number of overlapping p/o is compared against the union of the two sets. They are simpler, yet efficient, ways to express such a concept
* In 3.2 an extension is provided for literals with language tags but nothing is said about literals using data types
* The experimental results are only granted one page, leading to a compressed and vague description of the outcome and some results omitted for an acknowledged lack of space.
* As an evaluation measuring the description completeness reached from using high-quality links produced in the context of OAEI would have made more sense to me. The goal of the metric is not to assess the quality of the links but instead to measure how much additional information is gained from them. Using link sets from OAEI and observing the gain would have been a more focused and informative experiment.