Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO

This paper aims to measure the quality of five famous knowledge graphs by many criteria, and proposes a framework to recommend the most suitable knowledge graph for a given setting. As a survey, the authors focus on the quality of knowledge graphs and give lots of metrics in multi dimensions. The detailed description for every metric, including definition, discussion and formalization, has directive significance for KG evaluation. The five KGs are representative ones in semantic web domain. Comparisons among them help readers to understand knowledge graph well. Besides, the readability and organization of this paper are greatly appreciated.
Contrast to the previous version, the paper changes a lot in the aspects of organization, expression, metric and so on. The new version gives specific definition for data quality and provides solutions for former suggestions and improves a lot. The authors add many solid criteria such as “Accuracy”, “Trustworthiness” and so on. They also describe the criteria in detail and classify them. The five KGs are counted in the aspects of trips, class, entity and relation, and measured according to the mentioned criteria. However, there are still some comments as follows:
1. The figures are lack of further description. For example, in Figure 4, why there are much more media entities in Freebase? Why there are no organization entities in Wikidata or no biology entities in Freebase? It’s strange as the two KGs are cross-domain.
2. Although the paper proposes a recommend framework and detailed steps, it has no concrete example for how to use it. Giving a particular scenario, or listing the existing applications based on the five KGs will make the paper more practical.

The authors have significantly improved upon their original paper where they demonstrated many of the common features and differences between publicly available knowledge graphs. The number of measures used for analysis has reduced by one, but a number of additional statistics and more in depth explanations have been provided.

All of the metrics are now well described with explicit quantification which should allow a reader to reproduce the results exactly and apply the 34 metrics to any new KG.

Metrics such as ability to rank data, and existence/support of, e.g., owl:disjoint, could easily skew the result in favour of a particular data set with many ‘features’ without regard for quality, of course it would be up to the reader to choose a sensible weighting for such measures. This is a situation where the metrics now being valued continuously between 0 and 1 rather than simply true/false allows for better flexibility.

I feel that the gold standard is at the most risk of being subjective, while the metrics derived from it (completeness) are some of the more important values. I trust that the authors have truly chosen a suitable gold standard set, their descriptions would suggest so and the resulting values appear fair.

I enjoyed the expanded discussions and evaluations of each metric, in addition to the summary table of references. I would be happy using the information within this paper and being able to back up each decision and only ask the authors to perform a final spell-check, e.g., p5. firectly => directly.

I thank the authors for answering my previous questions and addressing mine and the other reviewers’ concerns.