Review Comment:
(1) Suitability as introductory text, targeted at researchers, PhD students, or practitioners, to get started on the covered topic.
Needs further work.
(2) How comprehensive and how balanced is the presentation and coverage.
Large amount of material, good coverage.
(3) Readability and clarity of the presentation.
Presentation needs improvement / further work.
(4) Importance of the covered material to the broader Semantic Web community.
Topic is important to the community.
Summary:
This survey article focuses on tools for linked data consumption. The authors define the concept of a Linked Data Consumption Platform. They propose a set of requirements that such platform should satisfy along with a set of evaluation criteria to verify if these requirements are addressed by a tool. The authors differentiate among different user groups for such platform such as LD-experts and non-LD experts (e.g. data journalists).
Strong points:
The problem of making Linked Data available and usable, in particular for non-LD experts is a timely and an important problem.
The number of papers and tools reviewed in this survey article is impressive.
Weak points:
The authors build the discussion in the article and the evaluation of tools upon the assumption that one linked data consumption platform should satisfy all requirements along the entire data consumption pipeline. Effectively, the requirements discussed by the authors are a union of functionalities of existing tools. In my view this assumption does not seem realistic or desirable. Specialized tools for specific tasks are likely to be more effective.
Although the authors intend to evaluate the requirements from the point of view of non-expert users, no real input from the users of this group is collected or analysed. I strongly recommend that the authors seek explicit feedback on the proposed requirements from real users in the target group. The requirements that are currently collected bottom-up from the existing papers and system functionalities do not necessarily reflect requirements of the users in this user group.
Some of the requirements presented in the article mix up the desired functionalities and the concrete methods to implement them.
Detailed comments:
The authors of this survey article have an ambitious goal of designing requirements for an LD consumption platform that should support non-expert users. The authors conducted an impressive amount of work, both in terms of the number of surveyed papers as well as the details of the tool analysis. Still, at the moment, the results of this work seems to be preliminary and need substantial further development to make the findings of this work really useful.
One of the main results of the analysis, although not particularly surprising, is that currently none of the surveyed tools fulfil all the requirements collected by the authors with respect to the LD consumption pipeline. On the positive side, I think that this result can be taken as a motivation and a first step to create a road map for development of better LD consumption tools in the future.
One of the main points of criticism is that the survey methodology does not fully support the described scenarios, in particular involvement of non-LD users. Whereas, according to the authors, the goal should be to obtain the tools that support different user groups, such as expert and non-expert users, the authors collect the requirements, or better say existing functionalities of tools developed by experts. Such requirements are not necessarily meaningful to the users (especially in the non-expert user group). Instead, to become useful for real non-expert users, the requirements should be collected by directly involving the users into discussions, and allowing them to evaluate existing and request additional functionalities.
Another problem is the assumption that one platform / tool should facilitate the whole LD consumption pipeline, which does not seem very realistic, especially if one takes the quality of the services into account. Furthermore, the requirements might have different priorities, dependent on the target user group. Which of the requirements are the most crucial? For whom?
Criteria in the 3rd elimination round of tool selection is indeed very restrictive (i.e. being able to load a large scale dataset).
Beginning of Section 4 can be significantly shortened (similar processes described in the literature do not need to be repeated in full detail).
In section 4.2.1, please state explicitly that you talk about dataset search. E.g. “Search user interface” “Search query language” etc. are ambiguous (one can think about searching data in a dataset).
“The ability of the users to precisely express their intent strongly depends on the user interface” – I disagree with this. The user interface is just syntactic means to express the intent; what is also needed is the knowledge of data, schema, etc. please rephrase.
Furthermore, there is an issue with the formulation of requirements; some of the requirements mix up the goals and the methods to achieve them.
Requirement 5: Query expansion is a method, not the goal. The goal should be improvement of recall.
Criterion 5.1 should rather talk about finding semantically related results (here again, query expansion is just one possible method to achieve this goal).
Requirement 7: Also mixes up functionalities (ranking) with features (links).
Requirement 8: preview based on vocabularies is in my view very specific. While preview is useful, vocabularies are just one specific way to provide it.
Criterion 8.3: Preview profile ?
Criterion 9.1 … based on automatic querying … - too specific
Requirement 10. Which quality indicators do you mean?
“which model a similar part of reality” – please rephrase
The scores (in particular the averages) in the tool evaluation are not useful for the reasons discussed above.
Overall, the weaknesses in the requirement description again reflect the main weakness of the paper – it is, in its current form, a collection of specific functionalities of existing tools rather than user requirements of the users in the target group.
| 
