Digests, snapshots, events, or cumulative gaze - what is most informative of user success and failure? A study of the foretelling signs of user performance during interaction with visualizations of ontology class hierarchies

Thanks author for the clarification; however, I am not convinced that the paper in its current form is fit for this journal. It is rather fit for a venue on information visualization, HCI etc., since its findings are rather general and applies to information visualization in general.

If you see the excerpts from abstract and introduction below, the connection to the Semantic Web is only in general terms and tangential. The contribution of the article is not necessarily connected to Semantic Web, and it is more general, namely gaze enabled user predictions. Outcome of this study of course would be useful for Semantic Web as well, but also for any other field that involves visualizations.

The article would be relevant for this journal, if it was dealing or applying the findings mentioned in the context of Semantic Web with its intricacies. Semantic Web research surely uses scientific outcomes from adjacent fields, such as software engineering, machine learning etc., but that does not mean it should publish/consider contributions that primarily fit and should be evaluated by these communities, just because it could be useful for Semantic Web in the future.

>> recognizing timely moments where users may potentially need intervention
>> Experimental results suggest that irregular gaze events are most informative of early predictions, while increased gaze is most often associated with peak accuracies. Furthermore, cognitive workload appears to be most indicative of overall user performance, while task type may impact predictive outcome irrespective of the gaze analysis approach in use.
>> The contribution of this paper lies in the new knowledge generated on gaze-enabled user predictions, its feasibility to inform the timing of visual adaptation, and the various implementations of predictive gaze analytics leading to differing predictive outcomes in the context of realizing future adaptative visualization systems for ontologies.

First of all, I would like to thank the author for the efforts solving the different concerns that aroused in the previous round. Some of the answers have solved the some of the raised concerns, but I'm afraid that the contribution of the current paper is not still mature in order to get accepted:

- When I referred to the definition of the prediction task, I was referring to the actual inputs used in each case and the actual effect and boundaries of the prediction. As I have understood it, the current approach would get the set of averaged features gathered in the latest (this is important) window, and would use them to predict just whether the current user is above or below the median of the performance metric at hand. In this case, what's the actual median of the users? given that they are not experts and are evaluating the validity and completeness of a set of ontology mappings, it would be interesting to know their own performance.

- Moreover, regarding the prediction task and setup, given that the performance is the value to be predicted, it is not surprising that the cumulative gaze information (therefore, the window with the most information about the whole process) is the better one, and that digests and snapshots using only locally temporal information behave worse. This prediction task election (along with the fact that it had been already proved in [39] with DL methods) jeopardizes the current contribution: what about predicting the outcome of the next subtask? The actionability of just predicting whether a user is going to be performant or not in a global sense every two minutes is quite limited, and, in fact, the author raises it as a potential limitation for short tasks. I understand that the user's behaviour can be quite different, this is, for example, first understanding both models, and then establishing the validity of the mappings all at once, or, on the other hand, validating one mapping at a time. I consider that at least some analysis of this should be done in order to increase the value of the contribution.

- Regarding the user characterization, the author states in the answer letter: "the results would not be very generalizable if a study were to require all participants to have an intricate technical background in SW technologies to complete, as such an expert user sample is not representative of the real-world users of SW technologies". In fact, it's quite the opposite, no regular user is expected to see an ontology directly (let alone establishing mappings) or to work with Semantic Web technologies. They have a steep learning curve (e.g., SPARQL is far from being trivial to be used even for people already trained in SQL).

Thus, given that the focus is on the analysis of the tasks itself, instead of proposing an adaptation method for ontology visualization, I'm afraid that, in its current state, I would advocate for a major review including at least a finer-grain analysis of the user's tasks, and, if possible (not mandatory), an analysis with users which should be considered experts.

Minor comments:
- In Section 7, the term task is used in an ambiguous way. It's referring to prediction task, but, please consider to use a different term to make a difference to the actual task that the user is doing (i.e., correctness is not an actual task).

The authors have thoroughly addressed all of the comments in my review of the previous version of the paper. Some minor details follow:
- In the Introduction the goal of the work regarding the comparison of four
different approaches to gaze analytics in the prediction of user success and failure should be emphasized in a separate paragraph. It is repeated several times in the Introduction.
- I find that the contribution of this work could be taken one step further and be applicable to other areas outside of the ontology domain. A brief mention on how this work could be adapted to other areas and tasks would be worth mentioning in the Conclusions.
- A very minor detail of the writing is that in the Iintroduction, Page 2, last paragraph, there are two "In an atempt to..." in a row. One of them could be changed.