Review Comment:
Authors tackle the challenge of studying the state-of-art RDF archiving systems. To do so, they first describe potential archiving strategies and query retrieval features. Then, they review current archiving frameworks. Finally, they make use of a prototypical benchmark for archives (EvoGen) to provide an evaluation of two particular systems (TailR and R43ples). Results show that (i) EvoGen is not reliable enough as a benchmark and (ii) the evaluated systems do not scale or do not provide complex retrieval functionality, hence they point to the lack of maturity of the area.
Although the paper is certainly timely, from my honest point of view, the contribution and novelty of the paper is rather marginal. First, the review of the strategies and queries are not novel, and are sufficiently covered in references 10, 11 and 31 in the paper, and the important missing reference:
“Y. Tzitzikas, Y. Theoharis, and D. Andreou. On Storage Policies for Semantic Web Repositories That Support Versioning. In Proc. of ESWC, pp. 705–719. 2008”
The review of current archiving systems may complement the review of other related works (e.g. 11), but it is far from being exhaustive. On the one hand, the review disregards the authors' categorization of queries and does not indicate which queries are then supported. On the other hand, it misses at least four related systems:
- Shi Gao Jiaqi Gu Carlo Zaniolo. Rdf-tx: A fast, user-friendly
system for querying the history of rdf knowledge bases. In
Proc. of EDBT, 2016
- Ana Cerdeira-Pena, Antonio Farina, Javier Fernandez, and
Miguel A Martınez-Prieto. Self-indexing rdf archives. In Proc.
of DCC, 2016
- Frommhold, Marvin, et al. Towards Versioning of Arbitrary RDF Data. Proceedings of the 12th International Conference on Semantic Systems. ACM, 2016.
- I. Dong-Hyuk, L. Sang-Won, and K. Hyoung-Joo. A Version
Management Framework for RDF Triple Stores. Int. J. Softw.
Eng. Know., 22(1):85–106, 2012
Finally, although it is interesting that authors reflect the difficulties of running current archives (besides the missing works listed above), the evaluation of the archiving systems is too narrow and does not help to gain further insights for future developments. Note that the decision of using EvoGen, a very initial and prototypical benchmark (i) is very arguable and may lead to unreliability (as pointed our by authors), and (ii) is not justified in the scale as authors only test a very limited number of versions and data sizes.
Other remarks:
- [35] is categorized as annotated triples and hybrid strategy.
- Figure 1 could be improved by moving “Type” to the right side.
- The review of the systems in Section 3 is unbalanced: some systems are evaluated in too much detail and some notes on the evaluation and performance are given, whereas authors omit the performance (such as x-RDF-3X), or the strategy (e.g. Dydra which, by the way, I believe is not fully materialized as stated in Table 1) of others.
- Is blank node enrichment similar to skolemization?
- References (and expanded meaning) of LDF and LDP should be provided.
- Are the principles in the beginning of Section 4 standard? A minimum review of related benchmarks both in RDF stores and other areas (e.g. databases) would complement the work.
- The number and position of tables does not help in the comparison of the systems.
- What is the raw size (in bytes) of the corpus?
- In the beginning of section 5.3, it seems TailR is not evaluated in all the section, but it is indeed tested in 5.3.3.
- Although TailR does not support full querying, simple subject lookups could be tested.
- The “diachron” queries are introduced in the evaluation but they are not mentioned in the categorization of queries.
- The discussion of results is a bit naïve. When authors state that the important factor for query performance is the shape of the query, it seems they actually mean the number of intermediate results, which is an already well-known factor in SPARQL benchmarking.
|