Sem@K: Is my knowledge graph embedding model semantic-aware?

Submitted by Nicolas Hubert on 07/24/2023 - 04:59

Tracking #: 3508-4722

Authors: 
Nicolas Hubert
Pierre Monnin
Armelle Brun
Davy Monticolo

Responsible editor: 
Claudia d'Amato

Submission type: 
Full Paper
Abstract: 
Using knowledge graph embedding models (KGEMs) is a popular approach for predicting links in knowledge graphs (KGs). Traditionally, the performance of KGEMs for link prediction is assessed using rank-based metrics, which evaluate their ability to give high scores to ground-truth entities. However, the literature claims that the KGEM evaluation procedure would benefit from adding supplementary dimensions to assess. That is why, in this paper, we extend our previously introduced metric Sem@K that measures the capability of models to predict valid entities w.r.t. domain and range constrains. In particular, we consider a broad range of KGs and take their respective characteristics into account to propose different versions of Sem@K. We also perform an extensive study to qualify the abilities of KGEMs as measured by our metric. Our experiments show that Sem@K provides a new perspective on KGEM quality. Its joint analysis with rank-based metrics offer different conclusions on the predictive power of models. Regarding Sem@K, some KGEMs are inherently better than others, but this semantic superiority is not indicative of their performance w.r.t. rank-based metrics. In this work, we generalize conclusions about the relative performance of KGEMs w.r.t. rank-based and semantic-oriented metrics at the level of families of models. The joint analysis of the aforementioned metrics gives more insight into the peculiarities of each model. This work paves the way for a more comprehensive evaluation of KGEM adequacy for specific downstream tasks.
Full PDF Version: 
PDF icon swj3508.pdf
Previous Version: 
Sem@K: Is my knowledge graph embedding model semantic-aware?
Tags: 
Reviewed
Long-term Stable Link to Resources: 
https://github.com/nicolas-hbt/benchmark-sematk

Decision/Status: 
Accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Erik B. Myklebust submitted on 06/Aug/2023
Suggestion:
Accept
Review Comment:

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data, (B) whether the provided resources appear to be complete for replication of experiments, and if not, why, (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.

Review #2
Anonymous submitted on 23/Aug/2023
Suggestion:
Accept
Review Comment:

The authors have addressed the majority of the comments raised in the initial review, thanks for that. There are only a couple of things left to be fixed/clarified (see below).

- In Section 4.2 there is a discussion regarding untyped entities for the metric Sem@K[base]. It seems that one might need to actually extend it to entities for which some types might be missing (?). For example, if the range of hasCapital relation is capital, and London is known to be a city in the KG, but not known to be a capital, the triple would be incorrectly treated as a semantically invalid prediction, right?

- In the newly introduced example in Section 7.4, it seems that the fact that Seattle is a city should also be specified explicitly to make it clear that the triple is semantically valid, otherwise this is not the case based on equations 4 and 5.

- Further proof reading is required, as the paper still contains some typos/inaccuracies, e.g.,
* p. 5. "For instance, assuming head prediction is performed on the given the ground-truth triple"
* constrains -> constraints (typo in several places in the paper)
* Spacing around Section 4.2 seems corrupted.
...