Review Comment:
The paper describes the tool ABECTO, a command line tool to assess the quality of RDF knowledge graphs in terms of completeness and accuracy. As described by the authors usually quality assessment is performed if a gold standard is present which is not possible in the case of Knowledge Graphs having such a reference dataset. Therefore, the authors propose to compare the quality of the portion of knowledge graphs that describe the same things, for instance, compare the quality between musician entities from different knowledge graphs.
Overall, I had a good impression of the work although I think this idea is limited since it provides possible missing data. Each dataset has its own characteristic that may be guided by a particular use case therefore in terms of completeness we will have data that in one dataset can be more complete than in the other but this is not wrong if the aim of building such dataset is not to make it more complete than others. The problem would be interesting about the consistency of the same information. In this case, the author may identify if the fact expressed in one dataset is consistent with respect to the others.
Your work may represent a potential impact since it is a follow-up of a previously impactful contribution - this is usually insufficient unless you can make a very convincing case that impact according to impact beyond your own range of influence will be had very soon e.g. either by using your tool in other research groups or in other collaborative projects.
I have more detailed section-by-section comments below.
Sec 1
In the Motivation section which should be re-named Introduction, I think you should be able to expand and give a little background and some statistics regarding the importance of assessing the quality of knowledge graphs and telling why it is important (thus including the motivation part). Which are the challenges and how some of them are solved by state-of-the-art approaches (mention the most important approaches)? Which are the open issues? Here is the part where you start introducing ABECTO. A reference to the tool would be necessary from the beginning such as a repository on GitHub where the tool can be accessed immediately without waiting to read the other sections.
Sec 2
A lot of other works are discussed and the focus has been given to works proposing a tool. Although the discussion is interesting because it highlights the tools for assessing and improving the quality of knowledge graphs which are not so many, I still think that there are other relevant works that are in the same direction that are missing. For instance, the work of Nandana et al considers more quality metrics and the comparison is made between different versions of the same knowledge graph over time [A quality assessment approach for evolving knowledge bases].
But the comparison with the state-of-the-art should pay more attention to other works as well, such as:
- Jeremy Debattista et al. 2020: Evaluating the quality of the LOD cloud: An empirical investigation (In particular, here you will find a very wide range of metrics applied in the LOD cloud
- Knowledge Graph Completeness: A Systematic Literature Review (in this work there is a very thorough discussion about the completeness quality dimension which is very relevant for your work)
- Zaveri et al. 2016: Quality assessment for Linked Data: A Survey
Sec 3
Regarding the requirements list, the integration of the quality results from other tools on the same dataset can be made possible in R4 by aligning the schema elements but I don't see how it will be possible with quality results extracted from other tools. In order to measure the same quality dimension we should be able to check if the same quality metrics are applied and if the comparison is made with the same reference dataset. Integrating quality results from different tools is not that straightforward.
An additional point in the requirements is to provide a user interface with a dashboard where people can visualize and check the quality issues. Second, scalability is another relevant requirement especially when you want to compare several datasets with each other.
Sec 4
"Further resources are provided for the use inside the metadata graphs. The properties av:relevantResource, and av:correspondsToResource, av:correspondsNotToResource are available for the representation of the belonging of a resource to an aspect and for mapping results."
-> can you explain this better? What is a relevantResource? I am trying to guess things while reading this sentence. I think you should provide all the explanations to understand all the details better.
You need to explain all the new terminology that can be easily understood, e.g predefined metadata graphs, default graphs or key variables.
Why using could instead of can? Aren’t the variables used by the processors?
I think that section 4.1 should be reorganized saying that you are considering several parts of the vocabularies, i.e., you are proposing a modularization of vocabularies and then you explain each module. In this way, it will be easier to follow. What is the connection with figure 4, figure 5 and figure 6. It is not just an example but this is the whole vocabulary. I would suggest to keep this in a separate section
Are the parameters in section 4.2.1 all mandatory?
Mapping Processor in section 4.2.3 do we need to know in the graph what should be there a priori? It seems that we need to construct all the possible mappings.
Sec 5
This section can be integrated with the previous section or can be inserted with a discussion section. Usually, one section contains more than one paragraph.
Sec 6
Why should we consider this workflow to be cyclic? Aren't we supposed to have assessed the quality of our knowledge graphs and then improve it? Why do we need to go in a cycle?
* What does it mean that the plan execution is triggered automatically? Does "provides data for the further process"' mean data for the next steps!!!
* What does it mean to extend the knowledge graph?
* "own" knowledge graph -> is that the default or do you mean smth else? -> rephrase
* change or addition of a value in another knowledge graph -> how is it possible to bring changes in another knowledge graph that we do not own? Additional information is needed to understand better what you mean by this.
* You should extend and better explain all the cases under the results analysis.
* Knowledge Graph refinement, how does this change with respect to the previous steps? Extend and explain all the necessary details.
Sec 7
Rename the section to something smth like USE CASE Applications
* rephrase -> For a correct interpretation please note, that in contrast to the other knowledge graphs
* regarding the Jaro-Winkler Similarity metric seems to be limited. What about a metric for number similarity? What about other metrics?
* Table 1 is not clear by looking at the columns which is the overlap between these knowledge graphs. It is not clear how this comparison is done. I think in terms of completeness and accuracy we need to know the details of the formula applied. What is the quantity kinds count?
* do not use phrases such as "hopefully" in scientific work.
Sec 8
I think that there are also other metrics that do not need a gold standard such as the metrics proposed by Debattista et al. I think even in this work you still need smth to compare to but I don't think we can use exact metrics to do the comparison since we do not have presented the same reality. Here we should consider the problem of the open-world assumption vs the closed-world assumption. It does not mean that smth that is missing should be wrong since the objective is different for different providers.
|