Review Comment:
The article “The effect of generalization on user interpretation of topical overviews” deals with the representation of topical overviews of documents to users by evaluating the accuracy, the diversity, and the time users need to process the provided information. The article is organized into five parts, the fist part with reviewing related research on providing topical overviews of documents on one hand, and user interest profiles on the other. Then, the authors explain how they generated the topical overviews, and the central part of the article is a user study of which the results are presented and discussed.
Overall, the topic of information design is very useful today. However, the final results of this research are quite trivial: (1) users benefit from being able to see specific topics (2) users do seem to find the generalized profiles easier to interpret when the clustering algorithm performed well. Why is this research useful? How can these results inform the design for tools that “that enable interactive exploration of social media streams and other user-generated content”? A poorly written article that lacks a rigorous argumentation causes these questions. Additionally, the article was submitted to the special issue of “Mining Social Semantics on the Social Web”. The “semantic” part of the article consists of an SPARQL query.
The article would improve by another round of proofreading, and the areas of improvement are detailed following. This article is not ready for a journal submission. I would recommend submitting is to a workshop instead.
Abstract
* What is your contribution and where is it needed?
Introduction
* “Interactional aspects do matter for real-world applications,…” – This is one of the main issues of this paper. What is the exact use case? When does user need to interpret the output of clustering, recommendations or search functionality? What is the application context? Concrete examples would be every useful.
* The question: “Creating overviews necessarily involves abstraction or generalization, but which degree of abstractness is suitable for a given task?” does not really fit to the goal: “We therefore conducted an experiment to test whether the incorporation of generalized topics into topical overviews has an effect on users who were given a task in which they needed to interpret the overviews.” – If the degree is in question, then different degrees (not only two) should be provided. However, it is still quite difficult to understand the concept of “overview” in the context of the paper. It is a very general term, and it needs a clear definition (as many other terms need too).
Related work
The related work section reads more like a list. It is not clear, why the authors review the introduced research and how this research informed their research. Additionally, more information was needed for understanding the described concepts, for example, “spreading activation with two pulses led to the best results” and “category-based method”. I would recommend extending the concepts that are used later.
Generation of topical overviews
The missing linkage to existing research was done in the “flat presentation” section. Why here?
However, in the “generalized presentation” section, the elaboration of the presentation methods is entirely missing. The description is very artificial and contains some mistakes such as a union instead of an intersection, a wrong index (e in e), and the URL “snorql”. The authors neither provide an evaluation of the defined metrics nor an explanation on the used visualization (a nested list). Thus, the whole user study is based on the hypothesis that content is not related to presentation” Salganik, Matthew J., Peter Sheridan Dodds, and Duncan J. Watts. "Experimental study of inequality and unpredictability in an artificial cultural market." science 311.5762 (2006): 854-856.
User Study
* The description of the user study lacks many-needed information. For example, that is the concrete task; a user wants to carry out? The existing description: “Does the addition of generalized topics to overviews help users to perform a task in which they need to make sense of the underlying user-generated content?”. The three hypotheses used concepts that are not described: accuracy and diversity. A definition of the context is needed. These definitions are given in the results section, but this should be introduced earlier.
* The argument that a task-directed experiment is needed since it has not done before is not sufficient. What insights provide task-directed experiments as opposed to others?
* The used expert finding scenario in the area of journalism should be explained and described much earlier in the paper. However, the whole setup does not seem to be very realistic since the search system is very simplistic.
Results
Please change the caption in Figure 2. The colors present the categories and not the classes (there are not defined classes). The results section is compared to all other section well described. However, for example, it is not explained why the Shannon entropy is a good measure of the diversity index. Please move the whole line of argumentation in a separate section before defining the hypotheses.
Discussion
The discussion shows another major issue of this study – the influence of the user interface design (the nested list) on the study results. This should be taken into account from the beginning, for example, different designs should be considered.
|