NORIA-O: An Ontology for Anomaly Detection and Incident Management in ICT Systems

Tracking #: 3502-4716

Lionel Tailhardat
Yoan Chabot
Raphael Troncy

Responsible editor: 
Cogan Shimizu

Submission type: 
Ontology Description
Large-scale Information and Communications Technology (ICT) systems give rise to difficult situations such as handling cascading failures across multiple platforms and detecting complex malicious activities occurring on multiple services and network layers. For network administrators and supervision teams, managing these situations while ensuring the high-standard quality of service and security of networks requires a comprehensive view on how communication devices are interconnected and are performing. However, the relevant information is spread across heterogeneous log sources and databases which triggers information integration challenges. There are several efforts to propose data models representing computing resources and how they are allocated for hosting services. However, to date, there is no model to describe the multiple interdependencies between the structural, dynamic, and functional aspects of a network infrastructure. In this paper, we propose the NORIA ontology that re-uses and extends well-known ontologies such as SEAS, FOLIO, UCO, ORG, BOT and BBO. NORIA-O has been developed together with network and cybersecurity experts in order to describe a network infrastructure, its events (user login, network route priority reconfiguration), diagnosis and repair actions (connectivity check, firmware upgrade) that are performed during incident management. A use case describing a failure on a fictitious network shows how this ontology can model complex ICT system situations and serve as a basis for anomaly detection and root cause analysis.
Full PDF Version: 


Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 16/Jul/2023
Major Revision
Review Comment:

The previous version of this submission received a major revision decision. This updated manuscript has at most gone through a very minor revision. The substantial shortcomings discussed by myself and the other reviewers have not been sufficiently addressed. As such, my suggested decision remains unchanged.

Review #2
By Valentina Anita Carriero submitted on 02/Aug/2023
Review Comment:

I believe that the paper, in its new version, could be accepted, since it addressed most comments by the reviewers.

Review #3
Anonymous submitted on 11/Aug/2023
Review Comment:

I thank the authors for their response and their revised submission. However, the main points I raised in my original review have not been addressed adequately. Specifically, these concerns pertain to:

1. the ontology's general description and purpose,
2. the ontology's technical presentation, and
3. the ontology's evaluation,

I will provide more detailed explanations of these concerns and elucidate why I find the authors' response and revisions insufficient in resolving the issues I previously highlighted.

# The Ontology's General Description

In my initial review, I asserted that the presentation of the ontology lacked precision and scientific rigor. Specifically, I wrote that "there are no concrete statements that could be tested or verified."
It remains dubious to me

- what kind of anomalies can be detected,
- what kind of analysis is supposed to be done in terms of root causes,
- and how exactly the ontology would help with either of these tasks.

The authors' response, unfortunately, does little to alleviate this ambiguity. The authors responded to this point by writing

> "Expanding on the use of the ontology for anomaly detection, the principle is that it sometimes enables easier detection through queries on the graph when integrating heterogeneous data directly provides a 'complete' description of the anomaly (which was not possible without integration)."

This response exemplifies the pervasive lack of clarity and specificity throughout the paper. The gist of what the authors write is that 'the ontology enables anomaly detection for heterogeneous data through queries.' However, they fail to substantiate this claim with concrete examples or detailed explanations of the underlying mechanisms. Given the ontology's relatively simple structure, comprising 225 mostly basic RDFS axioms, it should be feasible to offer illustrative examples of such queries to demonstrate the ontology's intended use. Yet, no concrete argument is made and virtually no details are provided in terms of how the ontology enables anomaly detection through queries.

In the same vein, the paper is full of similarly verbose formulations that ultimately have little substance and are not supported by convincing arguments. For instance, the opening sentence of the conclusion reads

> "we have presented NORIA-O, an ontology for representing network infrastructures, incidents and maintenance operations on networks [...]."

The main keywords here are "network infrastructures", "incidents", and "maintenance operations". However, the term "maintenance operation" appears only twice in the entire paper: once in the abstract and once in the conclusion. Even searching for the term "maintenance" results in only four search results: two of these are due to "maintenance operation", as just described, and two more are due to "corrective maintenance actions". Needless to say, the expression "corrective maintenance action" is also not defined or described in a more specific way --- nor is the relation between the two the expressions made clear.

Similar shortcomings pervade other essential terms, rendering the paper incomprehensible even with the most charitable interpretations of the terminology employed. So, in its current form, the paper falls short in terms of illustration, clarity, and readability --- namely the main aspects of an ontology description submission at SWJ. These shortcomings are particularly surprising considering the relative simplicity of the ontology provided.

# 2 Ontology's Technical Presentation

The authors write in their response that

> "R2 argues that we haven't axiomatized enough, [...]"

This interpretation is not correct. I asked for a detailed description of the ontology in terms of its axioms. Furthermore, I asked for explanations and clarifications of specific axioms and modelling choices that appear questionable to me. Unfortunately, the authors did not offer any clarifications or explanations, and the revised submission still does not include a sufficiently detailed description of the ontology in terms of its axioms (I have seen the additional paragraph on page 6. However, stating that three commonly used OWL/RDFS constructors are used in NORIA-O does not adequately address the issue). I reiterate that this is crucial since the authors assert that the ontology enables "reasoning" throughout the paper, namely in the sections 1. Introduction, 2. Related Work, 3. Methodology, 4. Formalization and Implementation, and 7. Conclusion. My inquiry about what kind of "reasoning" the ontology enables and how this plays a role in practice has also not been discussed.

The authors' response continues

> "[...] so it seems complicated for R2 to understand how the ontology works (how it helps solve the problems of root cause analysis and incident management) [...]"

This is also not quite right. Even though the ontology's design is not sufficiently described in the paper, the ontology is publicly available allowing me to inspect it. The issue at hand is not that the ontology is complicated; rather, it is about how a relatively small ontology --- predominanlty built using basic RDFS constructs, many of which are used in a questionable manner --- can effectively contribute to addressing a seemingly involved task of "Anomaly Detection and Incident Management in ICT Systems" (quoted from the title). So, I asked for more tangible arguments and concrete statements about both the design of the ontology (in terms of its axioms) as well as design intentions. More specifically, I inquired about six concrete modeling choices, such as the rationale behind not classifying a "CommunicationDevice" as a subclass of "Device." None of my general or more specific questions have been answered in by the authors' response and the new submission does not offer any insights in this regard either. Consequently, I remain skeptical about the ontology's use in practice.

# 3 Ontology's Evaluation

The assertion that

> "[R2] finds that validation by Authoring Tests (ATs) is too weak of evidence of validity"

is a misrepresentation of my main point. However, the authors are correct insofar as that I also have doubts about the provided evidence of validity --- a point I will come back to later.

My central point was the absence of any form of evaluation in the paper, despite a dedicated section labeled "Evaluation." I reiterate that "testing" and "evaluation" are not synonymous --- neither in software engineering nor in ontology engineering. While tests can validate certain aspects of a design specification, an "evaluation" should demonstrate a) the ontology's relevance within its intended context, and b) the satisfactory fulfillment of its design goals. The paper presents *no evidence* for either a) or b) and should not include a misleading section titled "Evaluation".

Coming back to my reservations regarding "validity". The authors did not address my observation that "the ontology only meets about 60% of the formal design requirements derived from the formulated competency questions." I continue to believe that it is important to clarify why this is acceptable and why the ontology would still be relevant and useful for its intended purpose.

# Overall Impression

The new submission only includes minor changes to the original submission. Unfortunately, however, the changes in the new submission do not address any of the shortcomings I pointed out in my original review. Moreover, the authors opted to stay silent on virtually all of my specific technical inquiries concerning peculiarities within the published ontology and its design.

Overall, I am not convinced that the authors have any intention of improving their work w.r.t. the concerns I raised. Consequently, I recommend the paper to be rejected.