Evidence of Large-Scale Conceptual Disarray in Multi-Level Taxonomies in Wikidata

Tracking #: 3337-4551

Atílio A. Dadalto
João Paulo A. Almeida
Claudenir M. Fonseca
Giancarlo Guizzardi1

Responsible editor: 
Guest Editors Wikidata 2022

Submission type: 
Full Paper
The distinction between types and individuals is key to most conceptual modeling techniques and knowledge representation languages. Despite that, there are a number of situations in which modelers navigate this distinction inadequately, leading to problematic models. We show evidence of a large number of representation mistakes associated with the failure to employ this distinction in the Wikidata knowledge graph, which can be identified with the incorrect use of instantiation, which is a relation between an individual and a type, and specialization (or subtyping), which is a relation between two types. The prevalence of the problems in Wikidata's taxonomies suggests that methodological and computational tools are required to mitigate the issues identified, which occur in many settings when individuals, types, and their metatypes are included in the domain of interest. We conduct a conceptual analysis of entities involved in recurrent erroneous cases identified in this empirical data, and present a tool that supports users in avoiding some of these mistakes.
Full PDF Version: 

Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Masaharu Yoshioka submitted on 15/Feb/2023
Major Revision
Review Comment:

This paper analyses the current situation of Wikidata about the relation types (instance-of and subclass-of). They point out that there are many incorrect uses of such relations from their perspective. They also make a system to identify them. The idea is interesting, but there are several issues to consider, mostly related to the definition of correctness of knowledge.

As the author mentioned in the last paragraph of page 1, it is common to use the same names (strings) for the individual and class types. To handle such situations, OWL has three levels of representation, OWL, OWL DL, and OWL Full. In OWL Full, it is no problem to use both relationships (instance-of and subclass-of) for the same name.

"For example, it is perfectly legal in OWL Full to have a "Fokker-100" identifier which acts both as a class name (denoting the set of Fokker-100 airplanes flying around the world) and as an individual name (e.g., an instance of the class AirplaneType). "


Therefore, it is inappropriate to say that the existence of the relationship for the same name with two relationship types (instance-of and subclass-of) at the same time is wrong when we consider knowledge description using the OWL Full framework.

However, there is no appropriate reasoning framework to handle OWL Full, it is preferable to have a framework that supports constructing knowledge that can be handled in the reasoning framework (e.g. tools for OWL and OWL DL).

Especially for section 4. "Analysis and Discussion", the author assumes that Wikidata should be a consistent knowledge base for all. However, this discussion could be a method for extracting an appropriate set of knowledge fragments considering the OpenCyc basic scheme.

This paper should be rewritten based on this understanding.

Review #2
By Filip Ilievski submitted on 19/Feb/2023
Major Revision
Review Comment:

This submission implements and evaluates two anti-patterns against the April 2021 Wikidata dump. The analysis reveals millions of cases where the anti-patterns occur, and can be captured automatically with relatively small queries. Further analysis shows that while Wikidata has a mechanism for distinghuishing class levels, this is not used in practice. The links of the items in Wikidata to OpenCyc provide a possibility for such anti-patterns to be avoided in the future by automated means.

1. Work on anti-patterns like the ones proposed here provides interesting insights that can help Wikidata's knowledge quality.
2. The two anti-patterns are meaningful and their analysis shows that they are prominent in Wikidata.
3. The possibility to use OpenCyc to flesh out class distinctions automatically seems promising to help with AP1.
4. The automatic support tool is provided as open source software.

1. The paper analyses two anti-patterns, while the introduction claims that prior work analyzed only one. This does not seem like a huge jump in the novelty, especially given that a portion of the analysis and the automatic support is only given for one of the two anti-patterns. This questions the originality of this work. Also, given that the mentioned prior paper has been a workshop paper by the same authors, I am wondering whether the enhanced version of it is not more suitable for a conference than a journal.

2. Understanding the originality of this work is further hindered by the lack of connecting this paper to prior work. This is an obvious omission, as the paper does not include a Literature review/Related work section, and this is generally absent from the paper. I think that at least four aspects should be discussed in such a section: a) the concrete relation to the previous paper on anti-patterns by the authors; b) relation to prior work on Wikidata quality, especially the ones by Shenoy et al. (2021) and Piscopo et al. (2018, 2019), both of which analyze the confusion between instances and classes in practice at scale; c) relation between your proposed automatic support and the growing list of tools developed/supported by Wikidata for editors; and d) in-depth overview of theoretical work on distinguishing tokens and types.

3. Related to 2, the paper needs to position its content much better, especially in the introduction. I especially expected more discussion on the relation between tokens and types here, from the perspective of linguistics and philosophy. Moreover, the introduction talks about anti-patterns as problems and meta-class formalisms as solutions, but the examples for the two are similar in the introduction and it is unclear how to distinguish the two in practice (the paper, in fact, shows that meta-classes are not used in practice). Finally, the introduction says that prior work has only evaluated a single anti-pattern - but I cannot find information about how many anti-patterns there are, how were these derived, and how complete they may be as an error analysis principle.

4. On a related note, the paper does not specify its scope, which makes the analysis performed seem somewhat ad-hoc. It would help tremendously if the authors could list and motivate their research questions prior to presenting the results.

5. It is disappointing to see that the computational method that was used in this work did not prove to be reliable. Out of the two queries in 3.2, the second one could not run with Stardog. And for the analysis in 3.4 and the support in section 5, AP2 is not shown anymore because of performance issues.

6. Section 3 leaves the reader with many open questions, some of which are addressed in section 4. For instance, the entity counts may or may not be meaningful, given that many of the classes that have anti-patterns in tables 1 and 2 are generally well-populated classes, like disease and gene. It could have been more informative to distinguish the entities based on their position in the anti-pattern, or to compare their AP occurrences with overall occurences in Wikidata. Moreover, in the analysis about genes on page 6, the authors mention that their data comes from multiple sources, but it would be great if we could know whether some of these sources contribute the most to these anti-patterns and understand the underlying reason for that if possible.

7. The automatic support is nice and simple, however, it would be good to see how would this be intended to be used. Editors already have many tools within Wikidata - is the idea that the WAPA tool will be integrated with the other Wikidata tools?

Review #3
Anonymous submitted on 03/Apr/2023
Major Revision
Review Comment:

This paper presents an empirical analysis of erroneous and ambiguous distinctions of types and individuals in Wikidata. The recurrent error-prone structures are defined as anti-patterns. More specifically, the paper addresses the analysis of two anti-patterns. The paper is an extended version of a short paper published in [3], which only covers one anti-pattern.

This work addresses an important and well-known problem in knowledge modelling that is the distinction between types and individuals. This has been the subject of proposals in different domains such as ontology modelling, conceptual modelling, formal ontologies and so on.

Overall, the paper is well written and provides resources for helping users to analyse the impact of new statements in Wikipedia, in terms of the anti-patterns. However, it suffers from several weaknesses that have to be addressed before publication, as a full paper.

First, the contributions and the originality of the paper should be clarified. In fact, the authors state that "the problems identified are instances of recurring patterns involving instantiation and specialization that were identified originally in [2]"; "some of these problems were originally identified by some of us in [2] and characterized in terms of a number of anti-patterns". So, the anti-patterns discussed here have been identified in [2]? The analysis is new here? It is a matter of updating the statistics of 2016 [2]? The contributions should be clearly stated in the paper (that should also be self-contained), in particular with respect to the previous work [2,3].

Second, it is not clear that the two anti-patterns are exhaustive in terms of erroneous and ambiguous distinctions of types and individuals in multi-level taxonomies. While the analysis is based on Wikidata, how could it be generalized/reproduced to other multi-level taxonomies?

Third, the related work is very condensed. It is in fact limited to some considerations in the last section. A more comprehensive analysis of anti-patterns in the literature and those that applies to the problem here should be included in the paper.