Review Comment:
Overview:
This paper presented the qEndpoint, a triple store that implements a differential-update architecture for graph databases, by combining a read-optimized main partition (HDT) and a write-optimized delta partition (RDF4J). The performance of the triple store is evaluated over the BSBM and WDBench benchmarks. The experimental studies demonstrate that the combination of HDT and the delta is efficient and does not introduce an overhead. Furthermore, when compared to other existing systems, qEndpoint is overall comparable with other endpoints while reducing memory and disk footprint.
(1)Originality: This work is an extension of two previously published papers. Compared to previous work, two additional evaluations are added to this study. These augmentations are sufficient for the paper to be considered as a journal version.’
(2) Significance of the results: A key aspect of such a study is to measure performance in terms of both read and write (insert/delete) behaviour. However, only the first evaluation considers the write-related performance, while for the remaining two evaluations, it’s unclear whether write-related query workloads are considered, and related discussion is completely missing. Specifically, I have several questions that require clarification with respect to the evaluation:
1. In the first evaluation with BSBM: What’s the rationale behind not including a naive HDT as an additional baseline approach? The evaluation seems to only use a naive RDF4J as a baseline approach.
2. Concerning the second evaluation of WDBench, qEndpoint achieves much better performance than others, however, the paper lacks a clear description of whether write-related performance metrics were measured. In addition, I’m a bit sceptical about the results as the comparison is not conducted under an identical environment. Instead, it employs a ‘similar’ setup and draws the results from another paper directly.
3. In both the second evaluation with WDBench and the third evaluation using Wikidata query logs, comparisons are made among gEndpoint, Blazegraph and Virtuoso. It's worth noting that in the second evaluation, qEndpoint outperforms others significantly, while in the third evaluation, qEndpoint only outperforms on roughly half of the queries. Could you provide an explanation for this observation between the two evaluations?
(3) Quality of writing: There are several instances where the writing lacks clarity and could benefit from improvement:
a. In Table 9, could you elaborate on how the ‘%outperf.’ metric is calculated? Does it correspond to the distribution presented in Fig.3? Please provide further details regarding the information contained in Table 9.
b. On page 11, you mention, “We report the results in Table 4,” without providing a context or specifying what results you are referring to. Specifically, it’s unclear what you mean by ”average time.” Does each query run only once, and then you calculate the average time for each query type?
c. On page 14, there is another vague description: “The results are presented in Figure 3 and Table 10.” It is unclear what results are being referred to in this context.
d. On page 9, you refer to finding “the anonymized git repository with the experiments at [21]”, while the reference [21] points to WDBench.
e. In your explanations of some observations, you mention “query optimization problems” (once on page 10, another on page 14). Could you elaborate on what kind of query optimization problem you are referring to behind these explanations?
Data file provided by the authors:
(A) The source code of qEndpoint is provided on GitHub, containing a well-organized README file.
(B) However, the configuration files for evaluations and experiment results are not provided, which makes it impossible to determine whether the experiments can be replicated or not.
|
