|Review Comment: |
The submitted article was reviewed according to the SWJ review guidelines, which state in this case:
"Full papers – containing original research results. Results previously published at conferences or workshops may be submitted as extended versions. These submissions will be reviewed along the usual dimensions for research contributions which include originality, significance of the results, and quality of writing."
The article describes the DART system for enriching Knowledge Graphs (in the article written as "LOD cloud") with new facts, which consist of new relations between already given Knowledge Graph entities. The DART system consists of two main steps: Firstly, potentially relevant relations are gathered from the Web and are used to build patterns. Those patterns are then used in the second step, where statements (with the potentially relevant relations between existing Knowledge Graph-entities) are extracted from unstructured text. Evaluations apparently show that statements can be extracted which are not yet in the Knowledge Graph.
== Good points ==
* The article addresses an interesting and promising research area (in the light that Knowledge Graphs get more and more important and interlinked).
* The article is well-written and no significant grammatical errors were found. Hence, the quality of writing is fine.
== Weak points ==
* The originality of the presented approach is very weak. The presented approach is not very sophisticated and the methods of many components have already been developed by others (see more details below).
* Also the results -- as presented in the evaluation section -- are not convincing (see more information below). The system was compared to other systems only to a very limited extent. The evaluation results leave open whether the approach works in real-world scenarios, e.g., when dealing with large amounts of texts and relations.
* The related work section misses important references. The article neither outlines important works about relation extraction nor about Knowledge Base Population.
In the following, we present some general aspects which need to be clarified by the authors:
* The article writes about "Linked Data entities" and, hence, gives the impression that entities from different data sources might get interlinked. However, as the evaluation shows, only new links within a data source (i.e., Knowledge Graph) are established. The role of Linked Data in the article is therefore marginal and has no direct effect. I would suggest that the authors do not emphasize Linked Data, but rather speak of single Knowledge Graphs which need to be enriched. The authors might also think about revising the title of the article.
* I'm wondering why the DART system is designed to find statements where the relation part is completely new, i.e., arbitrary and not fixed. I would argue that Knowledge Graphs often already contain a given set of relations (at least on the schema level) which can be used for finding new statements. Extending the schema level (T-Box, i.e., performing ontology population) is different from Knowledge Base population on the instance level.
* The article does not address the challenges which arise when performing relation extraction. For instance, relational phrases, as extracted by DART, might be ambiguous, and multiple relations might occur between an entity pair at the same time.
We now point out single items which were noticeable when reviewing the article and which need to be considered for further submissions:
== 1. Introduction ==
* "...unless it is fully grown and updated:" Many data sources (Knowledge Graphs) will never be complete, as new knowledge is arising all the time. Hence, the sentence should be modified.
* As the focus of the article is actually not Linked Data, but only relations within one data source, the authors might think about reducing the introduction part about Linked Data and instead speaking more about information extraction from text. The text does not state so far that unstructured text documents are needed from which statements are extracted from and that there are numerous existing approaches for relation extraction (with and without a grounding in RDF). Furthermore, mentioning "Linked Data entity sets" might lead to the misconception that the entity set for subject and object of a relation might come from different data sources.
* The example :India batsman :Sachin_Tendulkar, as mentioned as output of DART, is inaccurate in the sense that not the country should stand in the subject position, but the Indian cricket team. The authors might either need to change :India to :Indian_Cricket_Team, or write it as :Sachin_Tendulkar :playesFor :India. :Sachin_Tendulkar :worksAs :batsman.
== 2. Related works ==
* In total, this section lacks pointers to important related works regarding
** OpenIE tools (ReVerb, NELL, OLLIE, WOE, ClausIE, etc.)
** methods for relation extraction in the Knowledge Base-context (see, for instance, the works [1-2], which are very similar to the presented approach in the article)
** prominent tasks and tracks such as the TREC Knowledge Base Population track and the Knowledge Base Acceleration track.
* The section contains a relatively long description of the ReVerb approach. Such a long description is not necessary (Why was ReVerb chosen among the OpenIE tools?), unless the approach has significant similarities with the approach presented later. In that case, the differences need to be described.
== 3. DART - The proposed solution ==
=== 3.1 Preprocessing ===
* What are "collections of data"? Obviously no literal triples.
* A concrete example for all steps of DART (here of the class pairs and of the entity set cross-product) might be very helpful for the readers.
* Regarding the choice of taking exactly 25% of the cross product: What were the constraints regarding the complexity and regarding the quality of the relations? I would assume that this value depends on the actual use case.
=== 3.2 Pattern-Discovery phase ===
==== 3.2.1 Extraction of patterns ====
* Constructing patterns (with the words between two entity labels and counting the frequency) is nothing new and not very sophisticated. Related work to mention here is, for instance, the BOA framework .
* The authors need to state whether "total number of patterns" refers to the total number of unique patterns or not. Taking 500 as threshold seems to be quite high, as not all relations might occur so often.
==== 3.2.2 Clustering using paraphrase detection ====
* It remains unclear how LESK was adopted for the presented work: Are still Part-of-Speech tags used in the approach?
=== 3.2.3 Representative pattern selection ===
* It is not stated in the article why each cluster needs a representative pattern. Is only the representative pattern of each cluster used later for statement extraction?
* Whether the representatives of the clusters are valid, is apparently not evaluated. Hence, it is unclear how the single steps of DART perform.
==== 3.2.4 Keyword extraction and Prospective relation formation ====
* My suggestion would be to rephrase "keyword" to "relational phrase".
* Are sentence borders in the relational phrases considered? This is not stated, but has an effect on the results.
* Also the performance of the Datamuse API is not evaluated. Is it an issue when a class name is combined with a word?
* "identifying a few examples under each category" suggests that this method was performed not very systematically: There might be assessments which were accidentally wrong. Can the authors elaborate on this assessment?
* Table 1: The assignment of the categories which are part of a relation and which are not seems to be not intuitive and might depend on the considered use case. For instance, why is noun.person included but not noun.location?
* It remains unclear whether all relational phrases (of the clusters) are used or only the representatives. An example of a keyword might be helpful.
=== 3.3 Triple-Finding phase ===
* When concatenating the entity in D1 with the relational phrase, the phrase found in the text might slightly vary from this concatenated string, e.g., due to additional words. Please explain to which regard this aspect can be ignored.
* "phr contains rel" in Algorithm 2 is underspecified. Please indicate how this is done. Also indicate, how the representatives of the clusters are used.
== 4. Experiments and results ==
* The description of the experimental results in Sec 4.1.1. - 4.1.3 is quite similar, so that also one section about all experimental results might be sufficient.
=== 4.1 Billionaires and Companies ===
* The considered entity sets for this experiment and the following experiments are quite small. Does the approach also scale well if we have thousands or millions of entities per entity set?
* Why don't the authors extract all triples given the entity sets?
* Why are the classes taken from YAGO, but the relations from DBpedia? DBpedia itself might be sufficient to consider.
* "Due to time and resource constraints, the recall value could not be calculated in a similar manner." This is elusive. Recall would be relevant to report in this journal article and is feasible to measure to some extent.
* The gold standard is not very comprehensible. Where does the ground truth come from? Please provide the text documents or at least information about the used texts (such as URLs).
* Regarding Table 2: The results are not very meaningful: Only 22 triples are in the ground truth and only 34 triples were extracted. Does Table 2 contain only cluster representatives?
* The article talks about enriching existing LOD data sets (Knowledge Graphs) with new relations, but does not consider the linking of the relations to existing Knowledge Base relations. Please justify.
=== 4.1.2 Rivers and States ===
* The authors argue that more fine grained relations could be found with the DART approach compared to the existing DBpedia Knowledge Base. However, it should be noted that Knowledge Graphs such as DBpedia are not designed to provide too specific relations, i.e., the existing statements are often reasonable.
=== 4.1.3 Cricketers and countries ===
* Instead of dbr:England also dbr:United_Kindom or dbr:Great_Britain might be relevant.
=== 4.1.4. Comparison with ReVerb ===
* Many arbitrary relations extracted might be already covered in the Knowledge Graphs on schema level or are irrelevant for the Knowledge Graphs. How do you tackle that issue?
* The sentence "Table 6 gives ..." is hard to understand, please rephrase.
* There are many other OpenIE systems which can be used for a comparison. Why did the authors choose ReVerb?
* The article does not consider how the relational phrases (patterns) are aligned to RDF relations. Is this planned to be done manually? Is it an issue that similar and related relations were detected (e.g., founded, founder, owned, ...)? A comparison with Dutta et al. [1,2] might be more appropriate as KB Linking is also an essential part in RDF statement extraction.
== 4.2 Application - finding missing links in the LOD ==
=== 4.2.1 Dance-forms and States ===
* Please consider that DancesOfIndia and StatesAndTerritoriesOfIndia are YAGO classes, while you might have considered only relations of the DBpedia ontology (this is not stated in the article). Please indicate which YAGO version you are using.
* The authors come up with the relations "popular in; popular folkdance of; ...", but do not describe whether those relations are the result of some filtering method, whether they are the representatives of the relation clusters, and whether methods were implemented which consider the ambiguity of relations (relations between entities of fixed entity types might still have various meanings).
* As the authors do not provide data about the ground truth creation (abstracts on the website), the creation of the ground truth is hard to track.
=== 4.2.2 Allergenic foods and Diseases ===
* Given the provided, quite limited data on the website, it is unclear to what extent the data (i.e., statements about foods and diseases) is useful for real-world applications. An idea would be to attach more context to those statements, which indicate the reliability and accuracy of the statements (e.g., provenance information). An evaluation whether the extracted statements are identical to the statements in the retrieved text might be good.
== 4.3 Discussion ==
* Various items of the listing in this section are not surprise findings. To name a few:
* The first item apparently indicates that there are many more potential statements "out there" which are missing in the respective Knowledge Graph. This is not surprising and concrete numbers are missing.
* The fact that the authors only check those objects which are given for the relation and for the subject seems to be relatively trivial.
* The listing of the prospective relations which were gathered also via noun categories indicates for me that not only removing nouns per se is advisable; instead, exploiting the associations with the relational phrases seem to be good. Is this correct?
= 5.Conclusions =
* As indicated above, instead of non-grounded statements (retrieved via OpenIE), it might be more reasonable to directly extract RDF triples out of text (see, for instance, existing approaches such as [4-5]).
In summary, the presented article tackles an interesting research problem, but even if the authors respond to all comments, I fear that the research contributions of this work are not enough for a journal publication.
 Arnab Dutta, Christian Meilicke, Heiner Stuckenschmidt. Semantifying Triples from Open Information Extraction Systems. Frontiers in Artificial Intelligence and Applications, 2014.
 Arnab Dutta, Christian Meilicke, Heiner Stuckenschmidt. Enriching Structured Knowledge with Open Information. WWW 2015.
 Isabelle Augenstein, Sebastian Padó, Sebastian Rudolph. LODifier: Generating Linked Data from Unstructured Text. ESWC 2012.
 Michael Färber, Achim Rettinger, Andreas Harth: Towards Monitoring of Novel Statements in the News. ESWC 2016.