Review Comment:
This paper is overall somewhat interesting but the quality of the writing lets it down. I have made extensive comments below in regard to the language which I hope the authors can address.
The introduction was particularly difficult to read because some parts of the text were not clear. See comments below.
I have organised this review into three sections. The main comments of this review are in the first section. Other comments related to writing only are in the second section. The last section is a extensive list of typos I found while reading (which really got in the way of understanding the paper).
This paper should not be published without these issues addressed, but it should not be a major revision.
Questions and things unclear
----------------------------
Page 2 line 1: "there are cases where no custom client applications exist". Is this because such client applications have not been built or because you can instead argue that there will always been a need for a general purpose application?
Page 2 line 7: "Nevertheless, due to this heterogeneity, it is difficult to compare them". To compare what? What is "them" in this sentence?
Page 2 line 22: You refer to the KLM model but you do not introduce or explain what the KLM model is until page 7.
It was not obvious before Section 3 that this paper is reporting on the development of a new version of an existing tool called Rhizomer, and the new version is called RhizomerEye.
In Section 3, line 5, you say "and taking into account the potential improvements already identified". I do not know what you mean or are hinting at by this. Please be explicit.
Section 3, line 10, "This approach is motivated by...". What approach are you referring to?
On page 4, first paragraph, you say that the tool allows you for verify that a dataset conforms to the specified ontology. How do you intend users to do this? Wouldn't this be manual, time consuming and error prone?
For the third bullet on page 4, it might aid the reader to point out that the class autocomplete is shown at the top of the screenshot in Fig 1.
Page 5 paragraph 2 line 1: I think the new tool was called RhizomerEye not Rhizomer?
Page 6 paragraph 1: From the description give, it sounds as if the user interface design was entirely driven by how well it would score on the benchmark, which is not standard UX design. This is not standard practice and could result in a very poor user interface. This is discussed later in the paper but I believe that this potential problem should be acknowledged here and the reader referred to the later section for further discussion.
Similarly, on page 6, bullet titled "Facet Values Autocomplete": Is it the case that the BESDUI benchmark is just about reducing the number of steps required to complete a given task? I believe that UX designers would argue that there is more to UX design than that.
Page 6, bullet titled "Global Text Search": You say that the search is available from the entry page, but you have not previously explained what the entry page is. Is it the overview page? If so, please use consistent terms.
Page 7 section 3.2 line 2: The KLM model is defined for the first time here but it is referred to earlier in the paper. It would be helpful to either give a forward reference earlier to this section in the paper or define the KLM model earlier.
I had read about the KLM model before reading this paper, but appreciated the refresher of the definition. However, I think the definition of H could be improved (for example, what is 'homing' on the keyboard?). The style of definition given in the Wikipedia article is clearer: https://en.wikipedia.org/wiki/Keystroke-level_model
Page 8, you say that "In the next section, we check
if this also translates to a fruitful and satisfactory user experience involving real users". I think this point, that engineering the UI to result in good benchmark results could lead to an unsatisfactory UI, needs to be made directly earlier in the paper because I believe that this is the first point where it has been clearly commented on.
Page 8 Section 4.1: What is the point of this "Material" section. Could this section be integrated elsewhere to simplify?
Page 8 Section 4.1: You cite [24], but this is a citation for Rhizomer (the old tool), whereas the sentence refers to RhizomerEye which is being described in the current paper. Why would you need a citation for the tool being described within the current paper?
Page 9 Section 4.3: You mention standards ISO/IEC 25010 and ISO/IEC 9126. Please briefly explain what these standards are so that the paper is self-contained. Furthermore, if you do not make reference to these standards in the text that follows then why do you include a reference to them?
Page 9 Section 4.3: "They are subsequently translated to KLM: K for keystrokes or button presses, P for mouse pointing to a target, and H for homing the hands on the keyboard or other device are the operators." It would be clearer to rewrite this as "They are subsequently translated to the KLM model (see section 3.4)." That is, refer back to the earlier section where you define the KLM model, which itself would hopefully be made clearer (see my comment above).
Page 9 Section 4.5: All of the participants in the study were computer scientists, and therefore any conclusions in the study are only relevant to computer scientists. What about people who work in bioinformatics? Several people in that field have knowledge of semantic web or similar technologies (or maybe alternative technology choices such as graph databases). It would have been informative to have included participants from other such domains. Otherwise, it is quite limiting that the conclusions are only relevant to computer scientists.
The number of participants in the study is extremely small, and I would hesitate to draw any conclusions from such a small sample. A lot of space has been dedicated to describing a study including so few people.
Page 10 Section 4.7 paragraph 4: As you state in the text, "the scale in the X axis starts from 2, as there was no 1 in the answers of the participants to any TAM question." I find this to be confusing and make the diagram harder to read (especially because this is not marked on the diagram).
Page 10 Section 4.7 paragraph 4: Why does ease-of-use score lower than usefulness? Could this be because the UI was reverse engineered from the benchmark requirements? This process is not industry best practice for UX design. I think that the authors should be more direct about this being the core weakness of this approach and it is shown in their own evaluation.
Page 11, in Table 3 you report the mean and standard deviation. However, for such a small sample it makes no sense to calculate the standard deviation, and I would argue that it is totally meaningless here. In general, in statistics, it is well-known that you do not calculate the standard deviation for small samples.
Page 12 final paragraph: "people that had to perform a ... difficult task first ... led to ... differences in the total interaction time amongst participants for the same task." This is an interesting point, and in real life users typically do not start with the most complex task.
Page 15 penultimate paragraph: I believe that the sample size is to small to draw the conclusions described here. You say that "the analytical results show their validity for this kind of participant" but you have a sample size of one. I think you could only claim that the analytical the consistent with the experience of this one individual (which is a weaker claim).
At the top of page 16: I agree with your assessment that the times in KLM are too small for these kinds of application.
In the conclusion (page 16 line 6) you mention "the potential risks arising from using BESDUI to guide Rhizomer development". I would have liked to have seen a section dedicated to an in-depth discussion of these risks about using the benchmark to guide the UI development. I think this discussion should have been centralized in one section because using a benchmark to make the UI decisions could result in a very poor UI.
On page 16, I agree with the comments made in the bulleted list, in particular adding support for a search for similar instances and adding text boxers to the sliders because most slides on webpages include text boxes (e.g., on ecommerce websites, sliders for price ranges almost always include text boxes).
On page 16, line -3, you say that "User interactions logs have been analyzed". It was not discussed in much detail about the interaction logs in the main body of the paper, and although they were mentioned in the introduction and in a couple of places, it is not clear what the interaction logs were used for.
In the conclusion (top of page 17) you say that "the interaction steps performed to complete the tasks are very similar to those determined analytically". I don't remember this being discussed in the paper, or do you instead mean the number of interaction steps?
In the final paragraph of Section 5 (paragraph before Section 6 on page 16) you discuss the limitation of the study. If these are the key criticisms of the study (and I agree with these criticisms) then they should have been given more space for discussion than 5 lines.
Confusing wording
-----------------
Abstract line 3: "have already been identified as drivers of the advancement of different domains"
Firstly, it should be "as drivers of advancement in different domains".
However this somewhat grandiose wording is still confusing on first reading to a native speaker,
when I think what you are trying to say is that "Standard benchmarks have lead to advances in different domains"
Page 2 line 22: "the real-time required to complete each of the tasks using RhizomerEye, when they were able to accomplish it. Even though the time required to fulfill proposed tasks" would be clearer as "the real-time required to complete each of the tasks using RhizomerEye, when they were able to complete it. Even though the time required to complete the proposed tasks", i.e., always use the same term for a completed task otherwise it is just confusing. (Also note the omitted "the" before "proposed tasks".)
Page 2 section 2 line 1: "Given the challenges for user interaction with a growing space of semantic data, ..." This sentence was very unnatural to read.
I think you meant something like "Given the user interaction challenges when dealing with large amounts of semantic data, which is continually growing, ..."
Section 3 line 11: "...the three classical data analysis tasks proposed by Shneiderman [25]: getting an overview of the data, zooming, filtering, and viewing details on demand." It is confusing that you say there are three tasks but then list four things. It would make more sense if written as "...the three classical data analysis tasks proposed by Shneiderman [25]: (1) getting an overview of the data, (2) zooming and filtering, and (3) viewing details on demand.", i.e., number each task explicitly.
Page 5, bullet labelled "Fact Values Filter". The description of the behaviour of exclude filters was confusing on first reading, but then I notice that the screenshot clearly exclude label that will appear when the value is clicked for a second time. Perhaps this could be explained in the text.
Page 8 Section 4.1 line 4: "RhizomerEye was not permitted advanced knowledge of the particular instantiation defined in BESDUI." This sentence does not make any sense to me. Please can you rephrase, or explain more?
Section 6 line 2: I do not understand what you mean by "and beyond the results already reported in [9]".
Page 7 section 3.4 line 4: The phrase "are the operators" in the sentence "K for keystrokes or button presses, P for mouse pointing to a target, and H for homing the hands on the keyboard or other device are the operators." makes no sense. It would be sufficient to remove the phrase "are the operators" and change the sentence to " for keystrokes or button presses, P for mouse pointing to a target, and H for homing the hands on the keyboard or other device.".
When describing the study, Section 4.3 is written is present tense, whereas Sections 4.4 and 4.5 are written in past tense. The tense of each section should be the same (probably past tense) otherwise it is confusing.
On page 9-10, you list tasks 1, 2, 3, 4, 6, 7 and 10. In the descriptions you use the terms client / consumer / user / she which all seem to refer to the same thing, i.e., a user. It would be clear to use the same term in each description so that it is obvious you are referring to the same concept (and user is probably the best term).
Page 10: In the table, make it clear that T1 is task 1, and so on, either in the main text or the caption because this was not obvious on first reading (and because you have U1 through U7 in the other table).
Page 11 line 1: "the benchmark of the different End-User Structured Data User Interfaces"
Please write this consistently. So either write "the Benchmark for End-User Structured Data User Interfaces" (which is the correct name written in full) or use the acronym "BESDUI". In general, it is confusing to keep switching between the acronym and writing in full after an acronym has been defined the first time.
Page 12 line 8: Change
"participant 6 in Task 10, that did not view the match between the author of review and reviewer until much later than other participants."
to
"participant 6 in Task 10, who did not realise that author of review and reviewer referred to the same concept until much later than other participants."
The original sentence was difficult to read, hence I am suggesting this rewording. Note also the change from "that" to "who".
Page 12 line -1: "In Task 6 (see 5)" What do you mean by "see 5"? Do you mean "see Figure 5" or "see Section 5"? Please correct.
It would be better if some of the discussion on the outcome of the study in Section 4.7 was moved to the discussion in Section 5 so that it is not repeated because currently there is some repetition between these sections.
Page 14 line 2: "The difference in time between the participants was derived from the search strategy"
I did not understand what this meant on the first reading. It might be more obvious to write something like
"The difference in time between the participants was cased by the manner in which the users performed the search"
Page 14 line 11: "not reaching a 4 out of 7 (7 being the best score) in almost no case". The double negation here makes this difficult to understand.
Could you instead say it more directly as, for example, "scoring less than 4 out of 7 (with 7 being the best) in almost all cases"
Page 14 line 11: "with most scores in 6 on a scale from 1 to 7". This is quite confusing to read. I think you meant "with most questions scoring 6 on a scale from 1 to 7". If this is indeed what you mean, then please rephrase this.
Page 16 line 29: "to discern the one they needed". What does "the one" refer to here? Do you mean the concept? It is best to be explicit and say, for example, "to discern the concept they needed"
Typos and language issues
-------------------------
Abstract line 3: "and till recently" -> "and until recently"
Abstract line 4: "Building on top of the Benchmark for End-User Structured Data User Interfaces..."
I think you mean "Using the Benchmark for End-User Structured Data User Interfaces..." because you are not layering something on top of this benchmark or using it as a library, i.e., the original wording makes it sound like you are developing the next generation of BESDUI.
Abstract line 8: "To rule this out" sounds informal and "To avoid this" would be clearer.
Abstract line 8: "User Experience" -> "User Experience (UX)" i.e., define the term before you use the acronym UX for the first time. Similarly elsewhere in the paper.
Abstract line 12: "potential RhizomerEye improvements" -> "potential improvements to RhizomerEye"
Section 1 line 1: "Through open data efforts like the..." -> "Through open data efforts such as the...". IMO, "like" is to informal in written English and it is better to use "such as". Similarly elsewhere in the paper.
Section 1 line 3: "into an increasing adoption" -> "into increased adoption".
Section 1 line 3: "...user-facing applications based on semantic technologies [4]. In fact, in most circumstances, this is the desired result."
On the first reading, it was not clear to me what the last sentence meant. I think the following would be clearer:
"...user-facing applications based on semantic technologies [4], and, usually, this is intended."
Page 2 line 3: "or semantic data they are generating" -> "or for semantic data they are generating" (missing "for")
Page 2 line 5: "Consequently, multiple efforts to make semantic data exploration interactive and intuitive are being made" -> "Consequently, there are multiple efforts to make semantic data exploration more interactive and intuitive" (add "there are", add "more", remove "are being made").
Page 2 line 12: "a new semantic data exploration tool, RhizomerEye" -> "a new semantic data exploration tool, which we call RhizomerEye". Make it clear that you are naming it here.
Page 2 line 15: "during RhizomerEye development" -> "during the development of RhizomerEye". The original wording sounds strange.
Page 2 line 18: "user experience like learnability" -> "user experience such as learnability"
Page 2 line 24: "four times larger" -> "four times longer". Should say 'longer' instead of 'larger' because we are referring to time.
Page 2 section 2 line 5: "well-established practices of User Experience (UX) evaluation" -> "well-established practices for User Experience (UX) evaluation" (change "of" to "for").
Page 2 section 2 line 6: "based on the evaluation with real users" -> "based on evaluation with real users" (delete "the").
Page 3 line 2: "like the Intelligent Exploration of Semantic Data Challenge" -> "such as the Intelligent Exploration of Semantic Data Challenge"
Page 3 line 7: "approach BESDUI is based on" -> "approach that BESDUI is based on" (add missing "that").
Page 3 section 3.1 line 4: "as the 300 most common classes" -> "because the 300 most common classes"
Page 3 section 3.1 line 5: "related among them" -> "are related"
Page 3 footnote 2: Shouldn't there be some citation here for the BSBM, or a URL?
Page 4 line 2: "data like that generated" -> "data such as that generated"
Page 4 line 3: "or in any case to verify" -> "or to verify"
Page 4 line 6: "the following set of features" -> "the set of features"
Page 4 paragraph 3 line 2: "in the sense that" -> "because"
Page 4 paragraph 3 line 2: "they make it possible to indicate" -> "these feature make it possible to indicate". It is always more clear to make it explicit what you are referring to.
Page 5 paragraph 2 line 1: "Like in the case of Overview, Rhizomer’s features supporting the Zoom and Filter tasks make use" -> "As on the Overview screen, the Zoom and Filter tasks supported by RhizomerEye make use"
I think this suggested wording is much clearer than the original.
Page 5 paragraph 2 line 1: "the ontologies the data is based on" -> "the ontologies that the data is based on" (add missing "that")
Page 5 paragraph 2 line 2: "they are used to retrieve" -> "the ontologies are used to retrieve"
It is clearer to be explicit about what you are referring to.
Page 6 line 3: "just at least one of them" -> "or only at least one of them"
Page 6 line 9: "facet values, those not among the top ten." -> "facet values, i.e., those not among the top ten." (add missing "i.e.")
Page 6 line 11: "allows for reducing" -> "reduces"
Page 6 line 13: "like integer, decimal, or year" -> "such as integer, decimal, or year"
Page 7 section 3.4 line 1: "Along RhizomerEye development process" -> "During the RhizomerEye development process"
Page 7 section 3.4 line 18: "also computed in the case of RhizomerEye" -> "also computed for RhizomerEye"
Page 8 line 4: "thanks to a very complex user interface" -> "due to a very complex user interface"
It is only appropriate to use "due to" hear and not "thanks to" because you are expressing a negative. It is confusing to read "thanks to".
Page 8 line 6: "but thanks to a really simply user interface" -> "and is due to the tool having a very simple user interface"
Not strictly required but I think this sentence formulation reads better.
Page 8 line 7: "which just allows completing three tasks, the simplest ones" -> "which just allows completing the three simplest tasks"
Page 8 Section 4 line 1: "In this section, the study ... is described" -> "In this section, we describe the study ..."
It is difficult to parse such a long sentence as currently written.
Page 8 Section 4 line 4: "had to perform, as described next" -> "had to perform are then described"
Page 8 Section 4 line 4: "Finally, the procedure followed to conduct the study is drawn" -> "Finally, the procedure followed to conduct the study is detailed"
Using "drawn" here does not make any sense, so use "detailed" instead.
Page 8 Section 4.1 line 1: "BESDUI benchmark was chosen, as it" -> "The BESDUI benchmark was chosen because" (add missing "the", remove unneeded comma, 'because' is clearer than 'an')
Page 8 Section 4.1 line 3: "discuss queries in real terms" -> "discuss the queries in real-world terms"
Page 8 Section 4.1 line 3: "For the study, RhizomerEye tool" -> "For the study, the RhizomerEye tool" (missing "the")
Page 8 line -1: "performing the tasks was also registered." -> "performing the tasks was also recorded."
I think you meant recorded instead of registered.
Page 8 footer 3: "rrweb.io record and replay functionalities: https://github.com/rrweb-io/rrweb" -> "https://github.com/rrweb-io/rrweb"
Only the URL needs to be provided.
Page 9 line 5: "Apart from that, user interaction logs" -> "In addition to that, user interaction logs"
Page 9 line 6: "As rrweb.io records the logs as all events" -> "As rrweb.io logs all events"
Page 9 line 7: "and treated by a script to focus on the events" -> "and a script extracts the events"
Page 9 line 11: "scrolls are also considered, defined as the amount of scrolls users perform" -> "scrolling is also considered, defined as the amount of scrolling users perform"
Page 9 line 14: "For overall usability measurement, another two questionnaires are used." -> "For measuring overall usability, another two questionnaires were used."
Page 9 line 15: "twelve questions to measure user acceptance" -> "twelve questions designed to measure user acceptance"
Page 9 line 17: "designed to get a measurement" -> "designed to obtain a measurement" (writing "get" is a bit informal)
Page 9 line 35: "features the products should have and others that should not" -> "features that the products should have and others that the product should not"
Page 10 task 10: "Get Information about a reviewer:" -> "Get information about a reviewer:"
Information should start with a lower case i to match the other tasks.
Page 10 line 8: "RhizomerEye opened in a Chrome web driver" -> "RhizomerEye was opened in a Chrome web driver" (missing "was")
Page 10 line 11: "Besides, in order to" -> "In addition, in order to"
I think you do actually mean that the statement is true in addition to what you said previously, not that it is sufficient on its own.
Page 10 line 12: "problems detected with each task" -> "problems found with each task"
You would rather say "found" when it is a human that discovered a problem through experience.
Page 10 line 12: "participants fulfilled the TAM" -> "participants completed the TAM" (completed sounds much more natural to a native speaker)
Page 10 line 29: "This way, the maximum, minimum, mean..." -> "The maximum, minimum, mean..."
It does not make any sense to say "this way".
Page 10 line 29: "As additional information, the first 6 questions..." -> "The first 6 questions..."
Writing "As additional information" sounds unnatural, and it would be better to just delete it.
Page 11 line 3: "The tool with more effectiveness" -> "The tool with the most effectiveness"
Page 12 line 9: "Results in efficacy in use indicate" -> "The efficacy in use results"
Also, do you mean "efficacy" or "efficiency" here? Because these are different concepts.
Page 14 line 10: "which indicate dissatisfaction" -> "which indicates dissatisfaction" (missing s)
Page 14 line 11: "(see Figure 7 are generally" -> "(see Figure 7) are generally"
Missing closing parenthesis.
Page 14 line 17: "achieved a mean of 73,33" -> "achieved a mean of 73.33" (use a period not a comma)
Page 14 line 18: "meaning of achieved score" -> "meaning of the achieved score"
Page 14 line -1: "yellow colour" -> "yellow" (you can delete the word "colour")
Page 15 line 1: "as the NOT they required".
I would prefer to say "as the NEGATION they required" but it is okay as is.
Page 15 line 2: "selected stroboscopes product feature" -> "the selected stroboscopes product feature"
Page 15 line 4: "Figure 2, And appears" -> "Figure 2, the AND appears"
(I am assuming that AND should be in capitals here because it is so elsewhere.)
Page 15 line 7: "Even though in the definition similar was stated accompanied" -> "Even though the definition of similar was stated as accompanied"
Page 15 line 8: "the other did not have" -> "the others did not have" (missing s)
Page 15 Table 4 caption: "Showing the results" -> "Shown below are the results"
Page 15 line 27: "when Task 6 is excluded, they are about" -> "when Task 6 is excluded; they are about"
You can either use a semicolon here or a full stop but not a comma.
Page 15 line 30: "Finally, where the biggest deviation" -> "Finally, the biggest deviation" (delete "where")
Page 16 line 21: "It was disturbing for users" -> "It was confusing for users"
I think that "confusing" is more correct here than "disturbing" because disturbing primarily means to make one feel worried or upset, so I think you just mean confusing.
Page 16 line 25: "limitations come up from" -> "limitations arise from"
Page 16 line 25: "number of participants and" -> "number of participants in the study and"
Page 16 line 9: "overfitting Rhizomer to BESDUI" -> "overfitting RhizomerEye to BESDUI"
The new tool is called RhizomerEye so I assume you meant RhizomerEye here and not Rhizomer (which is the name of the old tool)?
Page 16 line 10: "minimal amount of interaction steps" -> "minimal number of interaction steps". It is correct to say number here.
Please also proof read the references section in the PDF (not just the BibTeX file) because there are several typos.
|