Clover Quiz: a trivia game powered by DBpedia

Review SWJ "Clover Quiz: a trivia game powered by DBpedia"

The paper presents a trivia quiz game for mobile devices that uses questions which are generated from facts of the DBpedia knowledge graph.
There is a number of publications on generating quiz games from Linked Data in general and DBpedia in particular. Compared to existing work on that topic the author claims to provide a superior question generation approach. The game supports multiple choice questions. The questions are generated in a beforehand off-line process.

The question generation process has 4 steps: Domain specification, Data gathering, Category expansion, Category annotation.
The Domain specification step demands a manual definition of SPARQL queries, which are tightly bound to the DBpedia schema (or shapes). Listings 1 and 2 apply raw data triples (properties from the dbp: namespace - these are created from the infobox keys if (since 2016-10: if and only if) no mapping exists and hence are language dependent) which according to the DBpedia best practices should be avoided for high quality data. In the Data gathering step data is queried from the DBpedia endpoint and stored to a JSON file for each entity. The Category expansion step is described to be done via a recursive script, though it is already included in the SPARQL query given in Listing 1. Indeed, instead of a recursive script a simple SPARQL query would be sufficient. The Category annotation step is unclear, it seems to map categories to a proprietary taxonomy, which is nowhere described and motivated, is seems to be a simple removal of the dbc: namespace.

The remaining of the paper describes how questions are generated and weighted. Questions are generated from the entity files and question templates, which again are (hand-crafted?) JSON files. Entities are ranked according the a popularity score based on in- and out-links. Since pagerank values are originally provided by DBpedia, it is unclear why these are not used.

The paper concludes with a lengthy chapter on the backend server and mobile app, statistics on game usage and lessons learned.

In this paper I can not recognize a valuable contribution for the Semantic Web community. The extent of application of semantic web technology is limited: Linked Data technology is exclusively used for data gathering via hand-crafted SPARQL queries. As the data is queried it is transformed to a semantically weak JSON format. For that reason the finally generated questions are hardly connected to the original data, so that it would demand considerable effort to re-extract the facts the question was compiled from. From a Semantic Web perspective I would consider that approach an anti-pattern. Hence the application does not have exemplary character. The contribution is insufficient for an Application report in SWJ. Though, the paper touches some interesting topics, e.g. fact extraction from the WP categories, and unanswered questions, e.g. how can relations (or graph patterns) be (automatically) identified which are relevant for quizzes?

I strongly recommend to reject the paper for the SWJ.

The paper describes Clover Quiz, a trivia game powered by DBpedia.
An interesting application also showcasing how semantic web technologies can be used in edutainment/gamification. This is the main reason I suggest that the paper is accepted for publication with minor revision.
Quality of the application: I used the application and it functions as it is described in the paper. The number of downloads and respective comments also shows this.
I would suggest though one thing for possible future version:
Since the problem of the DBpedia latency is addressed (constructed graphs on local level), the author should consider implementing an online/real time version of the game, in addition to the turn based one, by implementing a game room (since node.js is already used). This way the gaming experience goes to "another level", really challenging.

Wrt the second dimension, i.e. clarity and readability, especially conveying the key ideas of semtech in the application, I'm also quite satisfied. The paper is explaining quite well the way the technologies are being used along with the game concept and the underlying processes. I would suggest though the following minor improvements:
1. What does systematically querying DBpedia during the data gathering stage mean? I'd like the author to provide more information.
2. The second paragraph of the question generation section (4) is imo too detailed and makes the reader a bit tired. Could be shrinked.
3. An overall architecture diagram, showing not only the back end but also how the processes are being run and which components are involved, could further enhance readability.
4. Can the author describe a short example of a problem report?

This is an application report on Clover Quiz, a turn-based multiplayer trivia game for Android devices with more than 200K multiple choice questions (in English and Spanish) about different domains generated out of DBpedia.

Positive aspects of the app/report:
The application seems to be popular among users and also has good number of downloads. It is good to see a real mobile app based on DBpedia. The report largely covers the technical aspects from a development/game perspective and author has done a good job of presenting them.

Improvements required:
The area where the report is weak is when it comes to reporting existing research or reporting how existing research was utilised to guide some of the design decisions in the development of the game.

The language of the text also has, in many places, more liberty compared to strict research writing - for example "DBpedia constitutes the main hub of the Semantic Web"

I am not too convinced with the latency arguments represented by the authors-"DBpedia endpoint, hence latency is too high for an interactive trivia game, as reported in [13]." since there are so many benchmarks that show that triplstores can handle singificantly larger knowledge graphs and  offer sub-second query response time.

Data extraction process :
- there are few unaswered questions here including 1. mapping the textual keyword to the concept from DBpedia - what if you have multiple options, of no option available matching the keyword,
but synonym or similar concept exists? 2. domain specific file - the format of the file seems random, although DBpedia has irregular property patterns, there are at least some common ones 
offering the opportunity to use a few template across all the domains? 3. wikipedia has category chains and you have used it in your work-however how do you decide in terms of how far to go in terms of sub categories? 

The comparison with the pervious work is quite broad with the criticism as: "question generation schemes that are not able to produce varied, large, and entertaining questions." For example, There are many works on the topic of linked data based query answering including a series of workshops (QALD). Although the submission is an application report, it needs to be positioned within these works. Few examples include: 
https://qald.sebastianwalter.org/
https://www.sciencedirect.com/science/article/pii/S157082681300022X
https://dl.acm.org/citation.cfm?id=2557529

The existing research also need to be reflected in various decisions made in designing the game - for example, the data gathering phase, question generation, etc to show if it was influenced by one approach over another. At the moment it does not refer to previous works and uses quite a broad brush in its criticism of the existing work - although it seems parts of this work follows design/research principles of other games/applications.

I would have expected at least some new lessons learned from a significant development exercise. Many of the lessons learned are not new - for example, messiness of DBpedia. Authors can add further value by focusing on the ease of using DBpedia, performance issues while using in production environment and other developmental issues specific to using semantics/linked data/DBpedia. 

I also feel that there is/was an excellent opportunity to analyse qualitative feedback in terms of your 5k users/downloads.