Review Comment:
# summary
The article describes the TermIt tool for managing 'legal thesauri' using SKOS and UFO (upper foundational ontology), along with its application in a Czech building-domain use-case. The tool information has been provided clearly, and the linked resource is both available and functional. The overall impression is that this tool/system has been developed well and has achieved usefulness and impact in its use-cases. In conclusion, I recommend the paper be accepted following some minor changes to the presentation of information and providing additional details regarding implementation, impact, and evaluation as stated below.
(1) Quality, importance, and impact of the described tool or system (convincing evidence must be provided).
- The article clearly articulates the necessity of semantic disambiguation of concepts with examples from legal tasks in Sec.1 and Sec.2. IMHO, this only covers the problem, and does not provide enough insight as to how TermIt helps with these. Given the short length of a systems paper, I suggest removing the use-case in Sec.1 (viz. construction), and using Sec.2 to provide the scenario with example as well as how a tool might satisfy this OR how TermIt specifically satisfies the issue using SKOS and UFO. (Note: my comment only concerns providing information on how a tool would help with the highlighted problems. Authors should best decide what flow/structure they want to present.)
- Having worked on legal documents and vocabularies for some time, I can understand the importance and impact. The Sec.7 information on impact only mentions use in scenario from Sec.2 and two additional ones. IMO, the impact and use-case sections should provide more information on all three use-cases together and be more specific about how TermIt has changed practices and/or helped with the tasks. This will help get a holistic view on what/where this is being used, by whom, and what the actual impact is beyond stating 'it is adopted'.
- The stated results of TermIt being actively used by a significant amount of users is great, and definitely shows impact. Some more details would be nice to drive the point - for e.g. links to the use-cases or projects, especially detailing the use of TermIt; or if there is a public page about the webinars conducted; or if there are public resources being created using TermIt (footnotes 14 and 15 relate to this, but it can be highlighted as being more significant than mere 'users'). In addition, some information on how the usage has been so far in terms of whether there was any feedback collected about using the tool, issues/improvements identified, plans for incorporating them, continuous developments etc. The last ones relate to whether the tool is still planned on being developed or the work has 'concluded'.
(2) Clarity, illustration, and readability of the describing paper, which shall convey to the reader both the capabilities and the limitations of the tool. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess
- The paper is well written, understandable, and information links are provided to relevant resources.
(A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data,
- resources are well organised, and documented (which is an important indication of 'mature' work)
- The long term resource link provided is for the docker image, which is okay-ish, but it would better to also highlight the nice pages for installation and tutorials in the paper itself (or mention at the least) ; https://kbss-cvut.github.io/termit-web/
(B) whether the provided resources appear to be complete for replication of experiments, and if not, why,
- I confirm that I accessed the resources and they were executable and functional.
(C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and
- GitHub is fine, IMO. However, the repo will always point to the latest commit, without a way to identify what version was "described" in this paper. Ways to do this include: creating a release (with an explicit tag), uploading the tool archive to a repo (e.g. zenodo).
(4) whether the provided data artifacts are complete.
- Yes.
# other major comments
- A screenshot (or a figure with multiple) would be nice since this is a tool.
- Identifiers can be made more visually recognisable, e.g. Sec.4.3 can use brackets as (g2) or g2) or use a table.
- Sec.6.1 on tool comparison mentions discarding open-source tools more than 2 years old. This is bad practice IMHO. First, Why/How did the authors decide on 2 years specifically? Second, 2 years is too low of a bar to exclude something. SKOS is a standard which has been around for a while, which means there may be tools that have been developed for it, are relatively stable/mature, but were excluded from this list. Third, how many such (excluded) tools were there? Or was there no research on them at all - and only tools that were recently updated were looked at?
- In continuation, why were other similar practices and tools used for legal documents not considered? For example, individual topics as - SKOS management, or document annotation. The EU Publications Office does a lot of similar work on publications using metadata where such duplicate concepts can be easily seen (e.g. search for concepts on IATE https://iate.europa.eu/). They use VocBench. Similarly, there's SKOS Play that only does SKOS related documentation, and is commonly known/used. It would be helpful to understand where exactly TermIt fits within this ecosystem - does it replace all these tools, or it fits within them i.e. some other tool can be used for a feature such as documentation? This also relates to the requirements (F1-F5) described in the paper.
- Sec.5 Related Work is too small to be effective. Also Sec.6.1 provides more information on related tools. Perhaps Sec.5 contents can be integrated with Sec.6.1 to make relevant work more comprehensive as well continuous?
# minor things
- In the interest of saving space if needed to keep the 10 page limit: Fig.2 UFO types can be removed; The triple on pg. 4 line 24 can made inline or preferably just expressed as plain text; Sec.6.2 user evaluation details can be shortened
- "SKOS compliant" is an odd phrasing because SKOS is not a protocol or a specification, but a vocabulary description. If SKOS set out some requirements that TermIT implements, then the sentence can be understood as meaning these requirements are implemented correctly and the resulting output is 'compliant' with those requirements. Perhaps "SKOS compatible" or "SKOS adherent" may be more accurate?
|