Review Comment:
I thank the authors for their response and revisions. While some issues regarding clarity and presentation have been addressed neatly, some key concerns regarding the applicability, instantiation and generalisability of the schema are not addressed and even somewhat reinforced by some of the responses.
Regarding generalisability across supervised and unsupervised ML: again, the authors response (in a very generic way) is not elaborating, how this would be covered (e.g. see my remarks regarding the "Run" class). In supervised learning, an ML implementation may be used to train a "model" which can be run to produce actual "outputs" (eg labels of classified instances). In unsupervised ML, no model is trained but an implementation is run with certain parameters on a dataset to directly produce outputs (eg clusters which in turn may translate to labels). The response by the authors on this remark is not addressing this problem but only mentions that their model only covers the "training" stage (model induction) but at the same time reiterating their claims about generalisability across different types of learning.
The current schema actually is not reflecting on such differences between different types of ML not to speak of representing increasingly popular and important notions such as "reinforcement learning" or "transfer learning".
Even for traditional ML models, it remains unclear how handcrafted features and the complex engineering behind can be covered with the provided models.
The fact that the authors are not sufficiently responding to these points, nor are they able to point to actual instances of their schema which address such problems (what would be the best and easiest way to address any concerns). That is somewhat reinforcing the doubts I and another reviewer shared regarding the lack of applicability and adoption/use of the schema.
Another key problem is the distinction between the schema and instance level, which is not sufficiently addressed and surfaces in several parts of the schema and paper. The provided responses do not actually alleviate my concerns but are indeed reinforcing them. For instance, regarding Table 1, where mismatches surfaced in the original version through this strucutured table, they've now been embedded into the text by verbalising the content of Table 1. Same holds for my remark regarding the EvaluationMeasureClass: it remains unclear why the authors are stating that they plan to instantiate specific measures/values through reification (in this case) but include a dedicated instance for the case of hyperparameter(settings). The modelling problem is exactly the same, i.e. one wants to model metrics (evaluation measures or hyperparameters) and their values.
Most importantly, the paper still shows a lack of experience and lessons learned from actually using/applying the model, where such issues would have been uncovered and could be discussed and addressed.
The authors also seem to agree with the remark that "hasOutput" may also be confusing, given that here "output" refers to the model itself, but in traditional neural network settings, one would use "output" to refer to the prediction output of a model. However, in their response, they do not address how both should be modelled and distinguished.
In summary, the schema appears to only address the claimed contributions/claims at a very superficial level. Hence, it would be crucial to either improve the schema significantly (and evaluate it through extensive application to real-world ML scenarios) or to narrow and reduce their claimed contributions to better reflect what is feasible with the provided schema.
While this work has potential to provide a foundation for better modelling, interpreting and reusing ML models, there is still significant work to be done in order to actually facilitate adoption and use of this schema. I would expect both an improved schema which actually addresses such concerns, and some form of knowledge base of populated real-world instances to evaluate the applicability of the schema. Both should be an iterative approach since actual use and application of the schema will help the authors to surface and comprehend the practical issues raised by the reviewers (and beyond).
|