SPARQL federated query debugging tool

SPARQL federated query debugging tool
Marek Moos, and Jakub Galgonek

The paper presents a federated SPARQL query debugging tool. The proposed tool operates as a web-based debugging proxy, intercepting and monitoring query execution in real time across all service endpoints without requiring their modification. By capturing detailed execution traces—such as query requests, responses, durations, and endpoint interactions—it enables precise identification of bottlenecks and error sources even in deeply nested federated queries. A case study demonstrates the tool’s effectiveness in diagnosing query failures due to silent endpoint-induced query modifications.

Overall the paper is well written and easy to follow.

Many papers on SPARQL federation engines already used a proxy to monitor data transfer, number of HTTP calls etc, but it was in the context of benchmarking. I think it is a good idea to have a well written and maintained proxy able to do that.

The tool is available on Github and an online demo (https://sparql-debugger.elixir-czech.cz/) is working. The user interface comes with examples and is easy to use. I particularly appreciate the “bulk execution node” allowing you to see nested loop joins. The UI is very reactive allowing it to follow the execution in “real-time”.

Concerning the UI, I regret that subqueries are not formatted ie. the query is formatted as “...endpoint/chebi?query=PREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E+”. Not easy to see the exact subquery sent to endpoints when debugging.

The tool is very “UI” oriented. Sometimes we just need a proxy with logs for post processing. I don’t see in the paper or in the github documentation (https://github.com/iocbbioinf/sparql_debugger) how to deploy the proxy for such a context.

It is also not obvious to me how i can use such a proxy with federation engine such as FedX, Comunica, Fedup etc. Is it possible to set up the proxy for federation engines or it works only with services query typed in the browser ?

If i read the paper as resource paper, I have also few questions/remarks:

* On the impact, I think that tool identified and almost fills a gap. Federated queries are hard to analyze/debug especially when used with federation engines. The approach is not original, but can be useful.

* On reusability, there is no evidence of a usage by a wide community of researchers/engineers. Maybe the authors know that, but it is not written in the paper (just an interesting use-case).

* On the design&technical quality; the UI part is well written and seems to work. However, I wonder if the tool can be used in a federated query benchmark with hundreds of endpoints and a very large number of subqueries. The documentation looks very poor to me. Where are the execution logs stored ? How is it possible to post-process logs ?? I don’t think another developer is able to reuse this code. It is also strange to me that I cannot download and install only the “proxy” itself without the UI. Also strange to see that .vscode or .idea has been committed on github.

* On the Availability, the source code is available on github. Maybe persistent UrI (PRUL,DOI) should be available. We don’t know if there are any plans to maintain this tool for the next few years.

To resume, i think the tool fills a gap for the federated query processing, but several important question remains:

* Q1: Is it possible to use the tool really as a web proxy without UI ? If yes, how to do that. Can you provide step by step the documentation to do that ?

* Q2: How to do post-processing with your tool ? Can you write a documentation explaining how everything is stored in the proxy and how it can be retrieved by programs ?

* Q3: Can you explain how the proxy is built such that another developer can maintain it ?

* Q4: Can tell us who are the current users of the tool and if you have a sustainability plan ?

The manuscript introduces a tool for monitoring and inspecting the requests and responses that are exchanged between SPARQL endpoints when processing a SPARQL query with SERVICE clauses in a federated manner. The tool even covers cases of nested SERVICE clauses (i.e., where the evaluation of a subquery at one SPARQL endpoint involves reaching out to yet another SPARQL endpoint). The tool achieves this by replacing the SERVICE URIs in the user-provided query with URIs that point to a proxy server that is part of the tool. All requests are then routed through the this proxy server which collects relevant information for the graphical user interface of the tool.

The tool works well and is indeed very useful. I can clearly see how this tool can be tremendously helpful for developers who want to test their federation queries.

The authors host the tool on a public Website for users to try out. The source code is maintained in two GitHub repos:
- https://github.com/iocbbioinf/sparql_debugger_server
- https://github.com/iocbbioinf/sparql_debugger_component
which are well organized and sufficiently documented.

The manuscript provides an excellent description of the tool, including examples, a detailed explanation of relevant implementation aspects, and a brief case study that demonstrates a concrete, real-world use case in which the tool has been used to reveal certain issues that occurred during the federated processing of a particular query.

There is only one disappointment for me with the manuscript: I wish the authors had provided some experimental results on the performance impact of using the tool in comparison to not using it. With the tool, all requests are sent through the proxy server, which essentially doubles the amount of request messages and response messages being send around in the federation. It would have been nice to see some measurements on the effect that this has on the overall query processing times. Yet, I understand that the manuscript is meant to introduce the tool and not to evaluate the approach, and I also understand that the tool is meant for debugging, where a potential performance penalty is not really relevant. So, I am fine with considering such an evaluation as out of scope of the paper.

One last thing that I cannot resist to mention, as it has become a bit of a pet peeve of mine, is the improper use of the adjective "federated". There are many places where the authors mention things such as "federated queries", "federated SPARQL queries", "federated query debugger", "federated query debugger tool", "federated service endpoint", "federated query examples". None of these things is federated. A service endpoint or the authors' debugger tool does not have the property of being federated; instead, they are components of a federation of data sources / of a federated architecture that integrates multiple data sources. Likewise, a query can also not have the property of being federated; it may be mentioned as a query over a federation or a query for querying a federation, and the processing of such a query may be considered as federated (because the different endpoints take part in this process as autonomous members of the federation), but the query itself is not federated. And an example can also not be federated. I understand that several other authors have made the same mistake, but that doesn't make it right. So, I urge the authors of this manuscript to fix this issue in their text by using the adjective "federated" only in ways in which it actually makes sense.

Minor observations:

M1) Lines 16-18 on page 2 talk about "services that [...] are executed too many times" and, also, "a specific service may be executed multiple times". This doesn't sound right. A service cannot be executed; instead, a service may execute something (for instance, a query). I guess the authors meant SERVICE clauses or SERVICE patterns instead of services.

M2) Last line of page 3: ?ID -> ?id

M3) Line 47 on page 4 talks about "a nested service". Similar to point M1 above, this should be "a nested SERVICE clause" or "a nested SERVICE pattern" instead.

The paper introduces a proxy-based tool for debugging federated SPARQL queries. It intercepts and monitors query execution across multiple SPARQL endpoints. The tool provides real-time tracing, service execution trees, and a web interface. Users can use the tool to diagnose errors, analyze performance, and detect unintended query transformations without modifying endpoints.

The paper correctly identifies major problem points in federated SPARQL query execution: lack of transparency in error reporting, performance issues caused by inefficient query execution strategies, etc. The proxy-based approach is a practical and non-intrusive solution, which allows for interoperability with various SPARQL endpoints. It also provides detailed service execution tracing. The web application provides an intuitive debugging interface, provides the user with service execution trees, etc. Additionally, the authors demonstrate the tool’s effectiveness with a case study, highlighting the ability of the tool to detect and resolve errors caused by endpoint-specific query transformations. All of these are correctly identified, well executed and well described in the manuscript.

However, there are some concerns which need to be addressed. I've organized them below for readability.

Workarounds: Is the workaround for the case with the double / decimal variable in the UniProt / IDSM usecase implemented in the tool? Have other such problematic cases been identified, and can they be hardcoded as workarounds in the tool? Also, how would the authors make sure these workarounds don't lead to unintended behaviour in other use-cases?

Overhead: It would be useful to run a set of tests on different use-cases to compare the query execution times (a) without the tool, (b) with the tool (a simple run) and (c) with the tool with debugging switched on. This would provide insight into the potential bottlenecks of the tool itself, which would indicate areas of future improvement.

Correctness: The proxy-based interception approach sometimes modifies query execution paths, which may introduce unintended behavioral changes. A set of correctness tests, hand-in-hand with the overhead tests, would be useful to determine whether there are scenarios in which the proxy-based approach can lead to semantically incorrect results.

Scalability: What is the scalability limit of the Java Virtual Threads approach? A stress-test would be useful to answer this question and let users know if their use-cases can be debugged using the tool, or not.

Docker Image: It would also be good if the Docker image of the tool is provided as an already prepared image, on a platform such as Docker Hub (https://hub.docker.com), for instance.