|Review Comment: |
The paper describes the details of an ontology, supporting a marketplace for platform-as-a-service offers. The ontology is used to annotate offers and requirements and then perform matchmaking using a broker service. The paper is well written and easy to read, and it contains a lot of details that makes the reader get a proper understanding of how things work, especially concerning the modelling choices in the ontology and the technical architecture. This is on one hand the strength of the paper, but on the other hand also a weakness - the paper is very long. Even for a full research paper 43 pages is a lot, and I would also like to argue that this paper does not really fulfil the criteria of being a research paper, but should rather be put into one of the two alternative categories "ontology paper" or "application paper". More specifically, the criteria for a research paper are originality, significance, and quality. Although there is nothing wrong with the quality of writing here, I do not see many original research results, nor do I see much significance in the contribution to the research community - practical significance and contribution, yes, but scientific significance in terms of new important insights or methods, probably not. However, the two other categories of papers mentioned above are restricted to short paper (according to the submission guidelines around 10 pages as I interpret it), whereas the paper in its current form would certainly not fit there either due to its length. My best overall suggestion would be to split the paper into three: one ontology description paper focusing on the ontology as such, which could then be submitted to that track of the journal, one application paper describing the matchmaking process and the corresponding broker software architecture etc., which could then be submitted to the application description track, and finally, an online technical report, which could look much like the current version of this paper (or with even more details), putting everything together and giving detailed examples of how to use the model and the service.
It is with slight reluctance that I give this recommendation for the paper, because I did in fact enjoy reading it, and I think it shows nicely a practical case of developing an ontology starting from an ontology design pattern (as part of an upper ontology). However, even apart from the formal criteria of the journal the paper does not have a clear target group - who are the readers? For a full documentation of an ontology I would expect to go to an updated webpage, rather than a journal paper (that may be outdated already at the next release of an ontology version, in terms of technical details), hence, the paper is not really interesting as a piece of ontology documentation. There are however interesting details in how the ontology was developed, certain modelling choices, i.e. the consistent use of DnS, requirements and application examples, as well as the nice discussion on scalability, which are really interesting and would fit very well as an ontology description paper, targeting ontology engineers and researchers, or potential users of the ontology itself. Similarly for the case of how the ontology is applied and the architecture of the brokering service. Although I am not an expert in that domain, I would assume that this could make an interesting application paper, but readers of such a paper would most likely not be very interested in the development method and details of the ontology, but rather how to use it and what the service can do. So these are two very different target audiences. Finally, some technical details fit better in a living online document, rather than a paper, since they may be extended or even changed in the future.
Considering my recommendation above, I will not go into great detail in my comments regarding the paper content. Nevertheless, I have a few comments and questions that the authors may consider:
* What is the motivation behind the choice of OWL 2 DL? Why not a simpler OWL 2 profile, with better scalability characteristics? What specifics of OWL 2 DL is it that you cannot do without in this particular application case?
* The requirements presented in the paper are quite high-level, and I think they should be - you do not have space in a paper to go into details on this. However, some of the requirements do not seem very concrete and easy to verify, since they contain terms such as "easily" - how do you measure that? Furthermore, concerning your more detailed requirements, one obvious evaluation of the ontology, in addition to the structural methods presented in the evaluation section, would of course be to verify that it fulfils its requirements. So even if you do not list all the detailed requirements in the paper it would be interesting to hear how many such requirements you had, in what form they were encoded (e.g. competency questions or something else), and how you verified them. Along these lines you could also mention something on the rest of the ontology development process, i.e. did you use a specific methodology, who was involved, where there several iterations etc.?
* Do you estimate that the ontology largely covers this domain as it is now? Or do you see many extensions in the near future? In the actual ontology file there are some comments mentioning points of extension, however, the ontology does not seem to be annotated with any version number/version IRI, nor any other metadata, e.g. what set of requirements it covers or similar. What is your plan for maintenance and extension, and how stable is the current ontology? This is not really clear from the paper, nor the ontology itself.
* From the paper it is not entirely clear what kind of reasoning (i.e. based on OWL semantics) the ontology is used for, if any, or if the "reasoning" is only embedded into SPARQL queries? Closely related to this is also the question on why you need DUL, i.e. just for the generalisation capabilities and interoperability, or also for reasoning, i.e. using inherited axioms from DUL for consistency checking etc.?
* You do mention the reason for using an SQL-DB as storage, but only towards the end of the paper, and not in great detail. This would deserve a more thorough motivation, since the obvious choice would otherwise be a graph store, in which case you avoid all problems with having to modify the schema in the first place etc. Since this motivation comes late, the earlier discussion on the mapping becomes obscure to the reader, when the obvious choice would be to choose another backend.
* For someone who is not really an expert in the application area of this ontology, this paper also raises the questions how the work relates to the filed of semantic web services and service matchmaking? Potentially some of the related work listed in the paper stem from this area, but it is not so clear unless you have been following this research area closely. Also the paper does not make a detailed comparison between this ontology and the related Cloud4SOA ontology, but rather just discusses them at a very high level. In an ontology paper, I would expect a more detailed discussion of their respective coverage and reasoning capabilities.
* Finally a note on the mentions of DUL. Whereas in most places DUL is described nicely, and DnS is mentioned as the pattern, there are also some places where DUL is in itself referred to as a pattern, which I do not really agree with. I think this is merely a bit sloppy use of terminology, however, it should be revised if the paper is resubmitted.