Path based and triplication approaches to mapping data into RDF: usability analysis and recommendations

Tracking #: 3348-4562

Authors: 
Paul Warren
Paul Mulholland
Enrico Daga
Luigi Asprino

Responsible editor: 
Armin Haller

Submission type: 
Full Paper
Abstract: 
Mapping complex structured data to RDF requires a clear understanding of the data, but also a clear understanding of the paradigm used by the mapping tool. We illustrate this with an empirical study comparing two different mapping para-digms from the perspective of usability. One paradigm uses path descriptions, e.g. JSONPath or XPath, to access data ele-ments; the other uses a default triplification which can be queried, e.g. with SPARQL. As an example of the former, the study used YARRRML, to map from CSV, JSON and XML to RDF. As an example of the latter, the study used an extension of SPARQL, SPARQL Anything, to query the same data and CONSTRUCT a set of triples. Whilst there are difficulties common to the two paradigms, there are also difficulties in fully understanding the implications of each paradigm. For each paradigm, we present recommendations which help ensure that the mapping code is consistent with the data and the desired RDF. We also propose future developments to reduce the difficulty users experience with YARRRML and SPARQL Anything. Finally, we make some general recommendations about the future development of mapping tools and techniques.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Ehsan Emamirad submitted on 31/Mar/2023
Suggestion:
Major Revision
Review Comment:

Firstly, I would like to thank the authors for submitting their work to the SWJ 2023.

The paper provides a very good summary of the different techniques available to map structured data to create linked data. While the paper doesn’t necessarily posit itself to present a survey of these techniques, for the purposes outlined in the paper, the paper suffices to review two techniques namely YARRRML (manual mapping method) and SPARQL Anything (automatic triplification method) and conducts a user study with the main focus of analysing usability of each technique. The paper concludes by presenting relevant and useful recommendations and developments based on the preceding analysis in order to reduce difficulties experienced by users.

1. Originality
The originality of the paper is in its ‘somewhat’ comparison study and usability analysis (although I would challenge the nuance here and further argue that the paper can benefit from improving its comparative analysis – more on this below), in addition to its recommendations for further developments that aimed at reducing user difficulties when dealing with the studied techniques. Apart from this, and in terms of motivations and key contributions, the paper’s motivations are well highlighted and documented and are aligned with the research directions and focus outlined in the SWJ.

2. Technical quality and significance of the results
Overall, the paper has a sound technical quality and the results do seem to enhance our understanding of some of the state-of-the-art techniques that help map structured data to RDF. With that said, there are a few concerns that I believe hinder the significance of the results:

My biggest concern yet is to do with the notion of usability. It is not very clear how usability is defined and subsequently measured - the paper, e.g., discusses readability as one of the components tested with users but I would have liked to see a clear and overt discussion of these components, and as a suggestion, a table/visual further linking these usability/user experience components to the questions.

In terms of comparing YARRRML and SPARQL Anything, while the paper does a great job of explaining what each technique does and further discussing the usability of each, it, for the most part, doesn’t really provide a comparative analysis of the two - i.e., the paper presents a usability analysis of each method individually irrespective of detailing how each method might stack up against one another. To remedy this, I highly recommend a table summarising the themes extracted from analysing each method and then drawing practical implications (and maybe conclusions in the context of usability and user experience) against each set of themes.

A further concern I have with the technical discussion is that after reading the paper, I am left unsure which method is a better one; at least in the context of their usability. I understand choosing an appropriate method merely depends on the use case, so, perhaps a recommendation here is to discuss the different use cases that might best serve each of the methods.

Similar to my critique above, I would have liked to see some sort of triangulation between the user studies and insights available in existing literature, if any. This is important given the qualitative nature of the study which yields another source of validation from contemporary literature.

3. Quality of writing
Overall, the writing is easy to follow and understand however a few points to consider:
• Research method and questions could be summarised in a more effective way, e.g., using multiple visuals to detail the procedure.
• I understand the limitations here but going back and forth between figures and text to understand each point is quite difficult.
• The analysis of each theme is too descriptive, as it only provides a commentary on what happened during the experiments. Here, I would highly suggest a balance between key insights and commentary. Having the insights summarised in a table (as discussed above) can definitely help here as well.

Review #2
By Sergio Rodriguez Mendez submitted on 05/Apr/2023
Suggestion:
Minor Revision
Review Comment:

* Summary:
The manuscript presents an empirical usability study that compares two different RDF mapping paradigms: (1) path-based (focusing on YARRRML) and (2) triplification (focusing on SPARQL Anything).
The paper presented an analysis of the common difficulties found in the study, along with recommendations and proposed future improvements to reduce the difficulty users experience.

* Overall Evaluation (ranging from 0-10):
[Q]+ Quality: 9
[R]+ Importance/Relevance: 6
[I]+ Impact: 8
[N]+ Novelty: 8
[S]+ Stability: 9
[U]+ Usefulness: 9
[W]+ Clarity, illustration, and readability: 9
[P]+ Impression score: 6

* Dimensions for research contributions (ranging from 0-10):
(1) Originality (QRN): 7.7
(2) Significance of the results (ISU): 8.7
(3) Quality of writing (QWP): 8

* Overall Impression (1,2,3): 81.33
* Suggested Decision: [Minor Revision]

* General comments:
The work is solid. The paper is mature and good: easy to follow and understand.
Section 2 ("Related Work") covers the appropriate methods, techniques, and tools for mapping data into RDF.
There is a novelty in the usability analysis and study.
In general, I didn't find any critical points about the manuscript's content.
The authors can improve the manuscript by addressing the issues presented in this review and feedback.

* Feedback:
{
The only negative observation is that, at some point, it's difficult to follow many of the explanations in sections 7 and 8.
The reader must go back and forth to follow the arguments and reasoning when presenting the found issues from the user experience.
The authors did a superb job of capturing the essence of the explanations. Still, it takes time for the reader to follow the thought process.

- About YARRRML:
Perhaps, the main problems found in the YARRRML study are that the participants lack knowledge of how JSONPath and XPath operate (context node, tree structure traversal, and recursive descendent).
It is something out of the scope of YARRRML. If in the future, appears in the Internet landscape a new semi-structured text file format with its object model and inner datum rule access, RML (and hence YARRRML) would not change drastically since its engines will implement the support of this new object model and rule access akin to JSONPath and XPath.
How the inner rule access is defined (object model and syntax) is out of the scope of YARRRML.

- About SPARQL Anything over XML documents:
Recommendation #2 for SPARQL Anything in sec. 9.1 is not generalisable to XML documents with a composite model of textual content and elements such as the following:
```
<example>
text1
<element1>
text2 <element2 /> text3
<element3>text4</element3>
text5
</element>
text6
</example>
```

The authors should address this use case.
Additionally, although less important, what is the triplification model and how to access CDATA sections (and, for that matter, also comments and processing instructions) in XML documents?
This might be out of scope, but the authors should briefly include primary user considerations when accessing the aforementioned XML structures.
}

* About the companion resources: (“Long-term stable URL for resources”) — Title of the attached files: "Materials for mapping study: structured data to RDF"
(A) The attached files are well organised. Their content is clear and easy to understand.
(B) The provided resources appear to be complete for the replication of experiments.
(C) The chosen repository might not be appropriate for long-term repository discoverability. The authors should move the files to proper long-term repository storage.
(D) The provided artifacts (companion documents for the study in place) are complete.

* Minor corrections:
pag#02 / Sec.#2: "JSON and XML, and thence into the more user-friendly YARRRML" # hence
pag#08 / Fig.#5: it's blurred! Must replace it!
pag#14 / Sec.#7.4: "in question 3 (right-hand column of Figure 9)." # left-hand
pag#19 / Sec.#10: "Our study compared two very different approaches to data mapping to RDF" # approaches of data

Review #3
Anonymous submitted on 27/Apr/2023
Suggestion:
Major Revision
Review Comment:

# Paper

The paper presents a between-subjects grounded theory study in which subjects (n=19) were tasked with mapping data into RDF using (a) YARRML or (b) SPARQL Anything, respectively. The authors aim to provide a "usability analysis" and recommendations for users and for the future development of each of the two approaches. To this end, they reflect qualitatively on the experiments conducted with users and derive some recommendations.

(1) originality

The goal to compare path-based and triplification approaches for mapping data into RDF from a usability perspective addresses an important gap. The community has developed a large number of RDF mapping approaches, but it has put dedicated limited attention to the usability of these approaches in practice. Ref. [24] cited in the paper already provides a more quantitatively oriented comparison, but focuses on ShExML. The presented study appears to be the first that aims to compare path-based and triplification approaches and with a qualitative focus.

An experimental usability evaluation of RDF mapping paradigms could yield important insights in the cognitive processes involved and inform appropriate design decisions. This is highly relevant to avoid a widening gap between research and practice and important if the goal is for these approaches to be adopted in practice, rather than merely being developing as research artifacts.

(2) significance of the results

Whereas I find the aim of the presented study original and relevant, I'm less convinced about the research design and the limited insights that result from it. Sections 7-8 presents a narrative account of commonly observed errors for each approach and Section 9 starts off with recommendations and guidelines for users that are informative, but do not yield insights about the usability of the respective paradigms. Section 9.2 is the most significant in terms of results and summarizes recommendations for future development of YARRRML and SPARQL Anything.

Whereas analyzing common mistakes is worthwile as part of a usability comparison, I find it somwewhat disappointing that we do not learn anything about the relative differences between the two paradigms in terms of usability (neither quantitatively nor qualitatively) and that no hypotheses or theories are developed that could explain why/which aspects make the approaches more or less "usable". Finally, the definition of "usability" is fairly narrow and the results only tell us which errors were commonly observed for each of the two, but not how the "usability" of the approaches was perceived by users and how usable (intuitive, easy to learn, efficient, effective..) they were more generally.

(3) quality of writing

The paper is very clearly written and easy to follow.

# Materials

The authors provide accompanying materials in a data file under a stable URL in an institutional repository.

(A) The materials are well organized, there is no README file in the download (would be good to add that), but the information on the landing page is helpful and the data is easily accessible.

(B) The provided resources include the tutorials and questions (task descriptions) the authors used in their experiments. The "study" files describe the tasks that the participants had to perform in detail. However, I could not find the source files to be mapped and the partial mappings provided as a template to participants (experiments.zip that was attached to the Email to participants, according to the study file, does not seen to be available) - although these files are included in the paper as (somewhat verbose) listings (partially included as screenshots), they are not in the files - this makes it at least very tedious to replicate the experiments (sorry if I missed where those are in the repository).

(C) The chosen repository (the OpenUniversity's institutional repository) appears appropriate for long-term discoverability - a DOI and citation for the materials are available.

(D) Completeness of data artifacts: the materials only contain the tutorial and question documents used; the files provided to participants are missing and no empirical data or data documenting the analysis is provided (e.g., the authors mention that the sessions were analyzed using the NVivo qualitative analysis tool and that they performec a "grounded classification of observed errors", but it is not clear how - if possible, it may help to make the artifacts generated in this process available).

# Recommendations:

- Methodology: the methodology of the study is not described in great detail - Section 3 provides an "overview of the study", but the methodology (grounded theory) is only mentioned in passing in a footnote of the last sentence of the section. Exactly how the "grounded classification of the observed errors, informed by the participants' comments" was performed remains unclear. The authors only note that the recorded sessions were analyzed using the NVivo qualitative analysis tool, but this analytic process is not described and remains opaque. I would also recommend to mention the methodology in the Abstract, as I would have expected a more traditional quantitive empirical research design for a usability analysis until p.4).

- Sections 7 and 8 report on common problems subjects experienced. The authors present a lot of (excessively) detailed anecdotal/narrative accounts of which mistkaes were made by some/many/more than half/almost all/a particular participant(s), but without any synthesis and theorizing. In my understanding, the main goal of grounded theory as a qualitative method is the generation (rather than testing) of hypotheses and the construction of a theoretical model. Rather than a list of commonly observed mistakes, I would have expected a more in-depth discussion and theorizing about the how the paradigms affect the cognitive mapping processes that led to these common errors.

- I was expecting an in-depth comparison of the two paradigms in terms of usability, but could not really find this claimed contribution (cf. conclusions, which states that "Our study compared two very different approaches") in the paper. The authors do not characterize and contrast the relative merits and evaluate design choices of the two approaches from a usability perspective. What the paper presents is two largely separate discussions of common mistakes users make and recommendations on how to avoid them for YARRRML and SPARQL Anything. I would recommend to reframe the discussion and put less emphasis on the narrative account of observed problems and instead focus more on an in-depth discussion of similarities and differences of the approaches from a usability perspective (synthesized from the study), including hypotheses and possibly a theoretical model explaining them.

- The authors seeem to define "usability" as (absence of) conceptual difficulties experienced by the subjects - which they aim to infer from the errors the subjets make. They exclude other usability aspects such as ease of learning, efficiency of use (time and effort required to create the mappings), syntactic aspects etc. I understand that mapping/query languages are not self-contained software products and cannot be evaluated in quite the same way (e.g., they can be used in different editors, environments etc. which could also contribute to/impair usability) and a focus on conceptual difficulties makes sense. I would recommend, however to clearly define what you mean by "usability" and define the contributions, scope and goal of the study more clearly early in the paper. In this context, I would also usually expect to find research questions, hypotheses etc. - I understand that grounded theory as the chosen method aims for a more open-ended, exploratory research process than more tradititional methodologies and aims to generate rather than test hypotheses, but IMO it would still be important to know what the initial/high-level questions and the scope of the study are from the begining.

- "Usability" is not an absolute concept, but also depends on the context - particularly, e.g., the complexity of the mapping task and the experience level of the user. A powerful tool that has a steep learning curve may not be very "usable" to the uninitiated user, but very efficient and more capable than a more "usable" tool aimed at novice users. This is not considered. According to Section 3, the authors did ask participipants about their previous experience with relevant technologies. The paper reports this as demographic information, but I did not find any reflections on experience level in the discussion of results. Given that this information was collected, it would be interesting to know if the issues discussed in Sections 7 and 8 were "beginner mistakes" that are easily overcome once subjects learn the concepts or grounded in more fundamental issues in the design of the approaches.

- The missing elements in the solutions were chosen to "test participants' conceptual understanding of the mapping process" - I would have expected that this is picked up in the discussion of results somewhere and that we learn something about participants conceptual understanding (and the differences across approaches)..

- I would find an overview figure illustrating the experimental design (questions, conditions, tasks etc.) useful.

- The headings of Sections 7 and 8 are misleading. They do not actually report on "the user experience", as subjects were apparently neither asked about their experience, nor were many aspects that contribute to the user experience of a tool considered. Apparently, the users' experience was inferred from the observed mistakes subjects make. I would find it interesting to actually include a discussion on the "user experience" and also report on that if the respective data was collected; otherwise, however, the Sections should be named more appropriately ("common mistakes"?)

- It is unclear to me whether the large number of listings provide added value to the reader. They do make the paper self-contained, but 2/3 of the paper are largely dedicated to describing the tasks that subjects had to perform before we get to the core of the paper - which seems excessive, given that the analytic process is not described at all. I would recommend to reconsider this structure and possibly move some of the verbose scripts to an appendix or a separate document describing the research design in full detail. This should free up space to focus on a description of the research process, a more in-depth discussion, theorizing and synthesis of general insights in the main part of the paper.

- I would have expected a discussion of limitations. It seems that several aspects of the study design mentioned in Section 3 limit the insights that can be drawn and should be clearly stated as limitations (e.g., representativeness of subjects drawn only from Open University and from W3C groups, not randomized - i.e., subjects were free to choose the condition, limitations in experience that may explain difference in perceived usability and may hinder comparisons, the "great deal of interaction" of study designers with subjects during the experiment, which may limit the insights that can be drawn from the study, small sample size etc.)? I understand that internal validity is not as much a concern in grounded theory studies, but it is IMO still important to note these limitations.

- The code listings are pasted as images, partially in low resolution (e.g., Fig. 5). This makes them difficult to read (particulary in print). I'd recommend to add them as proper listings (the Latex listings package may come in handy). This would also solve the problem that text size, syntax highlighting etc. are currently inconsistent across the listings.

## Minor comments:

- some misplaced/missing commas (e.g., Introduction: "convert this to RDF for the creation of linked data, has stimulated")