Foundational Patterns Benchmark

This paper describes queries benchmark using the unified foundational ontology (UFO) and discusses the optimizations they help reduce during query retrieval. The authors make substantial improvements from the first version of the article, such as extending the foundational patterns with more queries (I would not say complex queries since they are not compared to SoA features on query selection and complexity), adding Virtuoso triple store (and now we see from the Section 7 that Virtuoso is better than the other triple stores) and a UFO validator using SHACL Rules. They also answered correctly to almost all of my previous key concerns, but the paper is still missing the crucial answer to this question on benchmark paper “How is this paper not yet another benchmark”
I appreciate the efforts of the authors towards making the paper convincing and of good quality. However, there are still remaining efforts to solve the following main issues:

*Motivation of the benchmark*: The authors mention that the novelty is that there is no existing benchmark based on a foundational ontology. However, the results based on query times suggested that Virtuoso worked better, and there are other studies showing the same conclusion. What makes foundational ontology and datasets really relevant than the DBPedia dataset and queries for example? Or the LDBC Social Network Benchmark? The reader needs facts and/or better conviction on the motivation. Many of the datasets mentioned in the paper are from the same institution.

*Queries selection and complexity*: There is a clear need to align your types of queries with existing SoA studies, as suggested by the editors in the reference. How complex are the new queries? I still can’t assess the queries. Please, provide a detailed review of the queries based on the features in that aspect.

*Evaluation*: Thanks for adding Virtuoso and generating more datasets (10 Million). What about Blazegraph? Or any other triple store that is present in SoA benchmark but not considered in this one? Any motivation here?
In future work, the authors mention to cover more SPARQL features with OWL entailment regimes. What are those features different from the ones already present in some SoA papers? Please develop this point or make it more clear to the reader.

As mentioned in the paper, the UFO-indexing approach is probably the most interesting aspect of this work. I would suggest starting from that and see how the indexing can be adapted to other existing benchmarks (e.g., would it be possible to convert existing dataset to a UFO-compliant and then use the same JAPO library?), so as to compare and position your paper better.

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing.

I appreciate the effort made by the authors to fill important gaps in the the article, particularly with relation to the size of the data used in the experiments.

However, I am still sceptical about the value of the contribution. The UFO-indexing approach is not a novel contribution and the benchmark presented does not seem to make sense outside the specific case of testing a number of triple stores with that. In other words, who else would need this benchmark? Future UFO users would just use one of the best triple stores in your results (e.g. Virtuoso), without bothering of performing the bench ark again, right?
Don't get me wrong, it is useful to know which triple store performs better with UFO-shaped data, it's just that the resulting benchmark does not generalise outside UFO and therefore is of limited use.

The claim that existing benchmarks do not pay attention to the shape of data seems not accurate. The shape of data is taken into account at the RDF level, with complex graph patterns, etc… The presented work has a point on generating an ontology-specific benchmark (and this is the most interesting part of the article) but this aspect should be generalised to other ontologies (shapes) and evaluated for its capacity of generating useful and high-quality queries, to be a contribution.

What is the take-away message of the article? The performance of the tested triple stores is comparable to previous research, nothing new on that side. The indexing mechanism makes the queries faster. OK, but this is not surprising (and already published).

I feel appropriate to leave a decision to the editors.

Thanks for the detailed reply from the authors.
All my concerns have been well addressed. So I recommend accepting this paper.