Visualisation of Ontology Changes and Evolution: A Systematic Literature Review

The paper provides a first comprehensive overview of ontology change visualisation tools using a Systematic Literature Review (SLR) approach. The paper reviews 28 tools, including both available and discontinued ones, and methods for visualising ontology changes, categorizing them into list, graphs, and statistics types. With only 12 out of the 28 tools remaining available for use today, the authors dive deeper and conduct a follow-up analysis on more details regarding the tools’ popularity, publishing venues, objectives, interactiveness, focusing trend, etc. This review serves as a valuable resource for researchers and practitioners seeking to select appropriate tools for visualising ontology changes and evolution, as well as for guiding future tool development.

Strengths:
+ The paper provides a first comprehensive overview of 28 ontology change visualization tools, benefiting researchers and developers in related fields, and contributing to the broader ontology evolution topic from a practical perspective.
+ Includes detailed search strings, inclusion and exclusion criteria, as well as a workflow for the Systematic Literature Review (SLR), ensuring potential reproducibility.
+ Organizes tools into categories based on their visualisation types—list, graph, statistics, and mix—yielding a comprehensive summary table.
+ Offers valuable insights for future tool development, including discontinued tools that offer lessons on what might need to be avoided in future directions, and presents a flowchart for selecting available tools.

Weakness:
- When evaluating the tools in Section 4, the authors used the terms “highly capable of showcasing ontology evolution” and “generally capable of …” repeatedly, but the criteria for defining the degree of capability represented by “highly” or “generally” are not provided.
- The authors mentioned “for evolution analysis, the tools are more often only presented conceptually and not implemented” and “the implementation of the presented tool was never available and the publication only showcased a potential approach for visualisation”. Here, the ones without implementation may be more appropriately categorized as frameworks instead of tools, and they should be excluded from discussion in many subsections in Section 5, or presented as a separate subsection.

Minor Issues:
- Page 4, line 16, “used We…” -> “used. We…”
- Page 4, line 34, two “visualisation” in the search string, one of them should be “visualization”
- Page 6, line 17, same issue as previous.
- Page 8, line 5, “that servetypess the purpose..” -> “that serves the purpose”
- Page 17, line 38 & 39, “Expendable Horizontal Tree” -> “Expandable Horizontal Tree”
- Page 20, line 36 & 37, “B3 the change navigator shows..” -> “B3, the change navigator, shows…”
- Page 20, line 39, “in othervisualisationsn/reports” -> “in other visualisations/reports”
- Page 26, Figure.27, the year axis should not be floats, “2002.00” -> “2002”
- Page 30, line 19, “DWAT [29] Including” -> “DWAT [29]. Including”
- Page 32, line 14, “number of tools 28 and even small…slitting…” -> “number of tools, 28, and even smaller…splitting…”

The authors have undertaken a thorough literature review of ontology visualization tools, categorizing these tools along various dimensions, e.g. static / interactive, list / graph. This is potentially a valuable service to the community. I liked particularly the use of the tf-idf metric to identify important keywords. In this respect, it satisfies the first requirement for review articles, i.e. suitable as an introductory text. Moreover, the coverage appears comprehensive and balanced. The presentation is generally clear, although I do make some comments about presentation below.

However, I do have significant concerns and believe that major revisions are required before the paper is of an acceptable standard to be published in the SWJ. My fundamental concern is with a lack of critical analysis; more analysis would make the paper of more significant to the semantic web community. I also have significant presentational issues.

ANALYSIS
Firstly, there is a lack of real analysis in the paper. The authors have followed mechanical procedures, which is useful, but I believe this needs to be augmented by the authors’ own critique of the various tools. The kind of issues which need discussion are: do particular tools fit particular use cases; what are the advantages / disadvantages of particular tools; are some tools more suitable for regular users, and some suitable for occasional users? What advances need to be made”

Moreover, you make comments without justification; this is a significant failing. The phrase “generally capable” occurs 17 times. On each occasion you need some justification. E.g. “LDT is generally capable of showcasing ontology evolution” – how do you know this. Similarly, “highly capable” occurs 7 times. Then you say (page 16/line 38) “Switching between the Protégé editor and the LDT plugin is made easy” – again, this needs justification.

You need to give your reasoned opinion of how the tools would be used, and their advantages and disadvantages. I am also lacking a view of how the tools have developed over time. For example, how do recently developed interactive tools differ from the original interactive tools. Importantly, you need to say how you see the future – what future progress is required in these visualization tools.

PRESENTATION
Much more care is needed with presentation. I include below a detailed list of issues, in order of occurrence.

One point which recurs refers to the phrases ‘publishing time’, ‘time of publishing’ and ‘time of publication’. You use all three. The second and third are fine. However, to me ‘publishing time’ sounds wrong. To me, this sounds like the time to get something published, not the date of publication.

Page 1
line 21 ‘been introduces’ > ‘introduced’
line 43 I assume should be 190,787, i.e. with comma not dot
line 48 ‘changes, and’
Page 2
line 11 ‘interest’ > ‘interesting’
lines 39-43; the last two sentences of section 2 are very confusing – what are you saying here?
Page 4
line 16 need full stop after ‘used’
The box (lines 31 to 35) implies there is a footnote 2. I can’t see this footnote.

Page 5
Table 1. Firstly, it is not clear to me what value the ‘exclusion criteria’ add – these seem merely to be the negation of the inclusion criteria. Secondly, I am unsure whether your search criterion is ‘A1 OR A2 OR A3’ or A1 AND A2 AND A3’. Thirdly, I would like to see a clearer link between these criteria and the search string on page 4.

Page 6
line 22 “In the second round, we only list the tools that changed groups” – what does this mean? Why do you show OWLDiff and AberOWL, when these were previously identified in round 1.
line 27 “missing 12 tools”; again, some further explanation is required here; I don’t understand what 12 tools you are referring to.
I think the previous two points relate to Figure 36, but this requires more explanation. Additionally, I see no point in putting this figure in the appendix (actually, it appears to be sandwiched in the middle of the references). It would be much better to put it in the body of the text, close to where it is discussed. Moreover, it would be much better to flip these bar-charts through 90 degrees, so that the keywords were horizontal and thus more easily readable.

Page 7
line 39 ‘other types of study’ – do you just mean ‘other studies’?

Page 8
lines 1 to 9. I suggest this needs some rewriting – it is not entirely clear to me. Also, line 5 has a typo “servetypess”
line 35 “The following tools are very similar”. The word ‘following’ is confusing here, implying that the tools are different from those previously mentioned. I suggest “These three tools are similar”.

Pages 8/9
Stylistically, it is a bit odd that at the beginning of section 4.1.1 you have a paragraph which lists the tools before subsequent paragraphs for each tool, whilst at the beginning of 4.1.2 you do not have an introductory paragraph. I don’t think you need that initial para in 4.1.1 – I don’t think it says anything additional.

Page 11, line 51
“Extending small seed ontology [38]” This occurs in a number of places in the paper, with a capital E and then the other words beginning in lowercase. This reads a bit odd – I don’t think it is actually the name of the software, is it? In any case, you need to explain their approach – what does this phrase mean? I don’t think the text on page 12 really explains this.

Page 16
figure 13 needs better labelling. The diff areas and partial diff areas need identifying. Also, you have labels (a) and (b), and (a), (b), (c) – you should use a different labelling for one of these sets.

Page 19
Figure 18 you have just cut and paste a figure from another paper, leaving the figure numbering intact.

Page 20
Figure 19 – why no comments on c in caption.

Page 22
line 33 “is shown that can be seen” -> “can be seen” (i.e. delete ‘is shown that’)

Page 23
Figure 23. I assume this has been copied directly from ref [48]. The labelling figures 6, 7 and 8 are not acceptable, e.g. say 23.I, 23.II, 23.III – but obviously don’t use (a) etc, because use letters within 23.II and 23.III.

Page 24
line 25 – what is ‘b’?

Page 26
Line 17 “This means that on the x-axis” – “On the x-axis”, i.e. delete ‘This means that’
lines 17 to 20, and figure 27. Since you are dealing with small numbers here, the four in 2017 could just be due to random fluctuation. I guess this could be considered statistically. More recently, maybe Covid has had an effect? For the figure, please remove the ‘.00’ in each of the years; this just confuses the picture. Still, it is possibly the case that nowadays there is less work on ontology visualization in general, and evolution in particular. I wonder, though, whether the focus has moved on to knowledge graph visualization?
Line 24. I assume here the number of citations is the number as of the time of writing, for a paper published in a given year, as shown in the figure. If that is the case, you should state precisely at what time the citation count was taken.
line 33 “much more” -> “many more”

Page 27
Figure 28 “(a) Publication year” – both a and b show publication year. The caption needs to be something line “(a) Publication venue of paper”

Page 28
Line 26. “Using text mining” This reads oddly. Is the problem that the software in ref[48] has no name? Maybe say “a tool which uses text-mining” and then, referring to it later “the text-mining tool”.
Line 36 “but the research into evolution visualisation over from 2013 onward.” delete ‘over’.

Page 30
Line 31 “we identify a trend …” Strictly, I don’t think you have much evidence of usage, rather more graph visualization tools are being developed over the years.
Line 44 “Only recently, from 2020 onwards, have new static tools been published” An interesting question is why this has happened. Is it, for example, because these tools have novel features which don’t lend themselves to interactivity?

Page 32
Line 29What does “tools without applications” mean?

This paper presents a systematic literature review of visualising ontology change and ontology evolution. It reviews in detail 28 tools and methods and investigates potential dependencies between tool availability, publication type and year.

As a survey paper, this paper is rated as follows in terms of the survey criteria put forward by SWJ:

1) Suitability as introductory text, targeted at researchers, PhD students, or practitioners, to get started on the covered topic.

Medium. The paper excels by diligently describing 28 tools in detail. As such it provides a good basis as an introductory text to this topic. At the same time, there is a lack of clear definition of terms (what is an ontology change? what is ontology evolution?), which would also be expected from a survey paper. Furthermore, the analysis of how publication type, timing and availability interact is of limited interest. And finally, a key element of survey papers is deriving concrete, substantial research topics for the field covered to give its readers' an intuition about the major key topics relevant for the community. This is by and large missing (except for a few paragraphs in Section 6).

(2) How comprehensive and how balanced is the presentation and coverage.

Medium. The survey has a reasonable covereage of tools which are presented in a balanced fashion. The fact that only one digital library was consulted, biases the survey towards covering tools published in that particulat library and potentially missing other tools.

(3) Readability and clarity of the presentation.
Medium. The paper is well structured, although the text contains several typos and could be rephrased towards a more academic style.

(4) Importance of the covered material to the broader Semantic Web community.
Mideum. The material will primarily be interesting to researchers working on information visualition, onloty/KG changes/evolution, and possibly the broader area of knowledge engineering.

Strengths:
* survey on an interesting topic
* good overview of existing tools

Weaknesses:
* lack of clear definition of the surveyed topics
* weak methodology (several details are missing, many limitations)
* RQ2 is onky weakly interesting
* lack of outlook of future research topics for the community
* sub-optimal text quality

Given the above, I suggest that the paper undergoes a major revision. While I am aware that, realistically, the methodology cannot be changed at this point, the paper can be still improved by clarifying some of its aspects and listing its limitations. Much can be done however in providing a better definition for the field, deriving key future research themes for the community and generally revising the quality of the paper.

Detailed comments:
* Abstract: " Our analysis provides a novel resource for selecting appropriate tools for visualising ontology changes and ontology
evolution, and will enable researchers and practitioners to select the appropriate ontology change and evolution visualisation
tools for their respective tasks." - this promise is not fulfilled in the paper, to the best of my knowledge. While a flowchart for selecting visualisations appears in Fig 35, the paper fails to describe what this is, how it was derived, who should use it etc.

* RQ2 investigates the relations betwen publication type, time as well as tool availability. While this RQ is somewhat interesting from an academic perspective, it falls short of addressing the need of practitioners that need guidelines to select a suitable tool for their task.

* p2/l30: "An analysis of the methodology to highlight the importance of the chosen approach." - it is challenging to understand what this contribution is and where in the paper it is delivered.

* conside closing Section 2 with a clear statement of the gap in the research landscape that this paper addresses (currently each paragraph ends with some indication of what is missing, and it would be great to give an overarching gap definition covering these at the end of the section)

* Section 3: there are several aspects of the methodology that are not clear:
** the setup of two consecutive search stages is rather unusual. It was also challenging to understand from the paper, as this overall strategy is not presented anywhere (e.g., before S3.1 would be a good place) but rather emerges while reading the paper
** what was the search time-span chosen?
** what publication types were accepted (e.g., conference, journal, workshop, book?, thesis?)
** "visualisation" appers twice in the first search string
** using Scopus as a single digital library is a limitation for an SLR. The authors argue that this is a limitation and justify it by claiming that it is challenging to extract information from Google Scholar. However, there are other digital libraries that allow structured extraction of search results such as ACM Digital Library, IEEE Explore and Web of Science (https://dl.acm.org/, https://ieeexplore.ieee.org/, http://www.webofknowledge.com/). Yet there is no account that these have been considered.
** the selection process of the paper relies on the opinion of a single author; a second author is only involved in unclear cases. To ensure rigour and reproducibility, selection and data extraction stages in SLRs should be conducted by at least 2 participants (even if only on a percentage, e.g., 20% of the data) and inter-rater agreement should be reported. This is missing in this paper.
** "snowball tools" is used through the paper to refer to tools identified through the snowballing stage. However, it is easily miss-read as tools that support snowball processes. The authors are encouraged to revise this phrasing through the paper.
** it is unclear how the analysis dimensions presented in Sect.3.3 were identified. These should normally be derived from the RQs. Furthermore, following is not clear:
*** publication year/type: what happens if a tool is described in several papers? which one is considered for this SLR?
*** Cited by: how was this number obtained and when? (we assume citation counts increase over time, so it is importnat to know at which point in time they were collected for this study)

* the definition of what is meant by ontolgy change and ontology evolution is only weakly covered. Instead of being a central part of the paper (as it is usual in survey papers that clearly define/scope the field/topic under investigation), only a rather vague and shallow definition of ontology evolution is presented at the end of Section 3. The authors are asked to define in more detail and with more rigour (i.e., by contrating to other, potentially different definitions from the field) what they mean with "ontology change" and "ontology evolution" as early in the paper as possible (even before related work).

* Section 6 is rather brief and should be expanded as follows:
** for limitations: it would be great if these could be clearly labeled to improve paper readability (e.g., "Only one digital libarry used", etc)
** the future work content is very limited for a survey paper, which typically suggests key lines of work in the surveyed community. Therefore, extensions and also a more clear structuring/labeling (e.g., "Temporal visualisation") is suggested.

* Section 7: is in essence a brief summary of the paper; it lacks an overview of the answers to the RQs of the SLR. Also, Fig 35 is never referenced or described in the text (unless I completely missed this), although it looks quite interesting.

Smaller comments/typos:
* p2/l43 "Section 3.1 shows the analysis of our methodology." sentence out of order (placed between sections 5 and 6) and not fully correct as S3.1 describes the methodology.

* p4/l7: consider "setting" instead of "use case"

*p4/l24: "to come up with" (rather colloquial) => to identify

* p4/p6 - footnotes 2 and 3 are not visible in the pdf.

* p7/l7: "of them" => thereof

* p7/l48: repetition of "used"/"use of"

* p8/l5: "servetypes"

* Table 2
** column name "Citated-By" => "Cited-By"
** although the caption mentions that the tools are ordered by "type of visualistaion first", it is not evident what is meant by this. Maybe vertical labels could be added to each group of tools in the table?

* p11/l37: "Clicking on these groups shows the explanation" - clarify: of what?

* p19/l1 and p21/l35: "is highly capable of showcasing ontology evolution" - on what basis was this statement derived? How shall "highly capable" be interpreted concretely? see also "generally capable" at p20/l39; "more static" at p21/l51

* as there is no section 4.4.2, it does not make much sense to have a section 4.4.1

*p22/l33: "is shown that": remove?

* p26/l29: "the colours further help distinguish between visualisation methods" - please clarify which colour shows what.

* Fig 28 caption is too shallow, it should be more detailed, explaining what a and b show.

* p27/l42: "Two out of these three tools are still available today" - please clarify which are these tools.

* p32/l20: "In any SLR, there are plenty of limitations and potential future work to be discussed. " - is this so? I suggest to remove this sentence as it is not based on any firm evidence.

* p32/l14: "even small numbers" => "even smaller numbers"

* p32/l23: prove => provide?