Ontological Foundation of Hazards and Risks in STAMP

This paper contributes an ontology of safety related concepts with an application to the domain of aviation safety. STAMP and FRAM are introduced as relevant causation models in the problem area. This paper formalizes two ontology modules within STAMP: SHRO and SCLHP. The authors have made these ontologies available online and include pointers. For the most part, the writing is clear, concise, and easy to understand. The authors also clearly document their ontology engineering methodologies and processes for developing the concepts.

For the most part, this work meets the requirements for descriptions of ontologies contributions in the SWJ. The justification for the work, including the need to formalize the concepts of hazards and risks, is well-defined. The authors also do a good job of explaining their choices of ontology engineering design principles. The running definitions and examples are also clear. The authors provide several pieces of evidence for the quality and relevance of the ontologies, including an example SPARQL implementation and demonstration of relevance to competency questions. Overall, I would recommend this ontology description paper for publication pending the following suggested improvements.

Broader/conceptual improvements could include:
- Extend the discussion and comparison with similar ontologies in this application area both in the background section and under “Relation to our approach” (page 17); this is quite important for communicating the value of the work
- Include evidence for specific advantages, for example in performance, that systematic or formal modeling choices made in STAMP confer; this would offer proof for why this treatment of “hazard” and “risk” is better than previous efforts in this area. The authors state that the lack of a systematic comparison of hazard and risk concepts with that of other ontologies is a major limitation of this work; this is significant since one of the criteria for descriptions of ontologies papers should make an effort to draw comparisons between the contribution and existing work. I think this could be improved with minimal effort however; an illustration of the entailments of other current hazard ontologies alongside improvements in STAMP would be a compelling step in this direction
- Add several “unseen” examples to ontology validation; the approach to ontology validation currently relies on a single running industry example presented in Section 3.1.1 on page 8 and summarized in Table 3. This is important because the value of this work lies mainly in its extension and formalization of existing concepts from the Common Ontology of Value and Risk, adding concepts for hazards and mitigation as needed in order to formalize industry safety models, with relevance to industry scenarios. It seems a bit circular to also use the same example that partially informed the competency questions for validation. I think this could be improved by adding just one “unseen” example, possibly from one of the domain experts referenced in the text. This could demonstrate the strength and flexibility of the modeling approach as well as its relevance to future software development for novel scenarios.

Minor/technical improvements could include:
- Reformulate competency questions; for example, I find CQ7’s phrasing unclear
- Separate concept and relation terms in Table 1 on page 9 and explain it in the text or remove it entirely, keeping Figure 5 as a better explanation of these concepts “in action”
- Move the discussion of related work in Section 9 up so that readers can situate the contributions of this work appropriately
- Fix Table 4 typo (“particioations”)

This manuscript was submitted as 'Ontology Description' and should be reviewed along the following dimensions: (1) Quality and relevance of the described ontology (convincing evidence must be provided). (2) Illustration, clarity and readability of the describing paper, which shall convey to the reader the key aspects of the described ontology.

The paper describes a hazard risk ontology that could help in analyzing accidents and modeling control loop failures. The ontology is modeled according to the System-Theoretic Accident Model and Processes (STAMP). The paper was submitted as an ontology description, a category of submissions in the SWJ that has its unique requirements that allow an ontology to gain impact in the community. One of these requirements is that the that the ontology is free, open, and accessible on the Web. For me this is a hurdle requirement. While the ontology is free, it is unfortunately not fully available on the Web. While the profile itself is available at http://onto.fel.cvut.cz/ontologies/stamp-hazard-profile/0.0.1 one of its constituents, https://onto.fel.cvut.cz/ontologies/page/stamp-hazard-and-risk is returning a 404. Also, the profile, while accessible on the Web, is not accessible through the OWL API (i.e. through Protege) as it uses special characters that can't be processed. Both files that are accessible are named ontology.ttl (or the chosen filetype extension), which is neither best practice nor useful when trying to load the ontology in an editor. The URI's used for the ontology are not persistent (e.g. no web3id is used), nor are they stored in a location that is likely to be available in perpetuity. While this could be a ground for rejection by itself, I continued to give the paper the benefit of a doubt. However, there are a plethora of further issues that unfortunately prevent the paper to be published at this stage. Some of the major one's are mentioned below:

- The profile ontology itself consists of classes the like of "appropriate control actions were provided but not followed". This is not a class, this is use case that includes at the very least the following classes, "Control", "Action", "Protocol" and an object property "notFollowed" or "neglected". This profile ontology consists only of classes of this nature, some of which are defined as equivalent to each other. There are no properties in this Profile. It is unclear to me what the purpose of this ontology profile is, but in its current state it has no purpose and it is wrong. There are even typos in some of the classes such as "actuator relatd factors". Also, in STAMP, everything is time-bound, even in the requirements and use cases mentioned in the paper, but time is not modelled in the ontology. This would invalidate any use case.
- The stamp control structure ontology at http://onto.fel.cvut.cz/ontologies/stamp-control-structure/0.0.1 suffers also from too many modelling errors to mention all of them here, but some examples:
-- None of the classes or properties include any rdfs:comments, i.e. they are essentially not useable to someone who wants to reuse the ontology.
-- There is a mixing of capitalization in classes, i.e. Person is capitalized, while other classes are largely small caps. While the recommendation is to use large camel case, whatever you chose, it has to be consistently applied.
-- There are many qualified classes that have no unqualified variants to inherit from or to be related to, e.g. "controller process model change" could be a ProcessModelChange that is a subclass of a ChangeEvent that is related to an agent controller.
-- There are classes named "type" in the ontology, e.g. hazard state type. While there can be a need for a class that also has a type of the same name, it is often a clear indicator of a UML understanding of ontologies. However, here it is unclear what the semantics of these classes are as there is no description. One thing is clear, though, a class "type" which itself is a subclass of "type" is certainly wrong.
-- There are many classes that are equivalent to each other, but have different labels and different positions in the sub-class hierarchy. It is unclear what the purpose is and they are most likely to be wrong. "stamp-object", for example, is a sublcass as well as equivalent class of control-structure-component.
-- The ontology does not import any established ontology, nor is there any mapping to existing ontologies. Mappings to PROV-O for properties like derivedFrom, Event, Action etc., to FOAF for Agent, Person etc., to the Time Ontology for the missing time relations, to SOSA/SSN for Sensor, Actuator, Process., to QUDT or another measurement ontology for Units of Measure and measurements.
-- there are no datatype properties in the ontology, which makes one wonder how values can be assigned for use cases. This in combination with the missing time and measurement aspects of the ontology explains instances such as "two u. s. a helicopters shut down" in the example instance at http://onto.fel.cvut.cz/ontologies/stamp-friendly-fire-example/current/d...
-- the ontology does not include any metadata nor does it define its own namespace prefix. There is no ontology description either.
-- the ontology is not registered in prefix.cc nor in the LOV ontology vocabulary.
-- The authors claim that the ontology is modularized, but neither of the modules can be used by themselves as they all import each other.

Apart from the plethora of modelling errors and issues in the ontology there are many major issues in the paper too, briefly, they are:

- There is no related work to related ontologies. Nor is there any related work of ontologies that overlap with certain aspects of the modelled ontology.
- The title and the paper in general assumes that the reader is familar with STAMP. That may not be the case for someone who just wants to use this specific ontology. Using the abbreviation only in the abstract and title is therefore problematic.
- There is no justification why the authors use an upper level ontology. And then, why chosing UFL and not more prevalent ontologies such as DULCE, SUMO, GFO or BFO. UFL seems to be only available anymore in a version that is published by the authors under their own namespace: http://onto.fel.cvut.cz/ontologies/ufo/current/index-en.html
- Why did the authors chose the SABiO ontology engineering methodology and not those that are widely used and established, Guarino/Welty's, Gruninger's or Gómez-Pérez methdologies.
- While claiming to follow the SABiO methodology, the authors did not define use cases. The requirements mentioned in the paper seem to be not grounded in use cases. There is essentially only one use case, which is the instance that is modelled.
- The authors claim that the ontology was validated by a successful instantiation of SHRO in a real world situation. There is only one instantiation, which is the helicopter shot down accident example instance which suffers from a plethora of problems (one was mentioned above), plenty of others exist such as "26 people dead" as an instance or "two f 15 fighter aircraft". If these are instances in a use case, no instance of this ontology can ever be compared to each other, rendering the ontology ineffective. These could be quantified, i.e. a FighterAircraft instance with two named indivduals for each of the involved aircrafts and 26 named individuals of Deceased.
- The validation of that one use case was done by two domain experts. How was the use case validated? Why only one use case that only covers a small fraction of the ontology. There should at least one instantiation of a class/property in the total of all uses cases.
- There is no evidence how this ontology is in use or how it can be used. Are there STAMP model instantiations in a different language, e.g. XML, UML or database instances? Can you provide a mapping from instances of those to the ontology? Can there be an extension to schema.org so that existing health and safety applications that publish hazardous events on the Web could use this proposed extension?
- The authors are encouraged to investigate the ontology design pattern initiative to see if some of the modelling choices could not have used established patterns.
- Figure 5 describes an ontology that actually makes a lot of sense, but none of the classes in that diagram that are yellow, i.e. Accident, Requirement etc. (and supposed to be part of SHRO) are actually included in any of the ontology files that are available.
- The references are messy, e.g. N. Leveson's "Engineering a Safer World: Systems Thinking Applied to Safety" book is cited at least 10 times.

Overall, unfortunately, the ontology in its current version is not suitable for publication in the SWJ.

The authors have done a lot of work responding to reviewer comments, and the paper is much better for it. The main thing is that is was easier to understand and thus to evaluate the quality and significance of what was achieved. I really wanted to accept this paper, but the further along I got, the less certain I was that the quality was sufficiently high. I have a lot of industry experience modeling risk for global financial institution. I built an ontology for risks, controls and processes and ingested large databases of incidents, risks, controls. They were converted to triples and are being used in production applications. As acceptance criteria for this paper, I do not require that the ontology be any better than was built behind a corporate firewall – for the authors have no awareness of it. However, I should have confidence that the proposed solution would be workable in a real-world situation. I do not have that confidence.

A separate matter is the degree of innovation compared to related work. Again, the arguments are not compelling. The main thing highlighted by the authors, which I take to be valid is that it places hitherto disparate models of risk into a coherent framework backed by a foundational ontology. This greatly increases the scope for interoperability in future production applications of risk data. I felt that the authors did a good job of that, as evidenced by figure 5 and the accompanying explanation. However, there are some problems. The amount of new work is very modest. Most of the model is based on existing work.

A worse problem is the fact that the link to the ontology (http://onto.fel.cvut.cz/ontologies/stamp-hazard-profile ) does not describe the concepts that are described in the paper. So I was unable to evaluate the ontology. I expected to see a handful of modular ontologies, one for STAMP, one for Common Ontology of Value and Risk, one for UFO all there in support of the main work of hazards that was new. I found nothing of the sort.

The worked example was a good one, but given it was very small, it should have been modeled in full in Turtle. That would achieve two things. It would show the full sophistication of the model and it would be easy to see what chain of triple we traversed by SPARQL to give useful answers.
Unfortunately, the SPARQL examples were very simple and did not demonstrate the usefulness of representing the complex network of relationships as a graph.

The controller part was more substantial, but it was not developed very much and did not show up in the CQs. Anyway, I found it to be a bit confusing.

I am not going to recommend that the authors re-submit this paper. However if they do choose to continue this work, I have the following comments and suggestions.
• The idea of Value is not germane to the example in the paper. I found it to be a distraction. I see that it is central to the Common Ontology of Value and Risk, but not to your examples or goals.
• There are a lot of abstraction that are not clearly useful. For example, moments and dispositions. For example, the paragraphs in the 1st column of p6.
• P6, C2, lines 39-41 is circular. How can risk be the relationship of describing risk?
• P8, C1 lines 25-45. The hazard here is very abstract, basically “the fog of war” which affects clear communication. This is starkly different from a hazard defined by birds near aircraft, which is much more specific.
• There is a huge disconnect between the numbered CQs. CQI-1 has nothing to do with CQ-1. In fact there is no introducing of what a CQI is vs. a CQ. Very confusing.
• The ‘regular’ CQs are widely varying in their specificity. They are all good a starting points, but some need to be fleshed out into separate questions. Looking at them more carefully, they are not what I would call a CQ at all. Table 2 is more like a documentation of parts of the conceptual model. Unfortunately, none of it appears in the owl anywhere.
• P11 C1 line46 A8: this should be one way logical implication, not equivalence. Some minor thing that contributes to a risk event is not necessarily a manifestation.
• The concept of vulnerability just came out of the blue, what part of the model is it in?
• Regarding the controller ontology… I was confused by the distinction between control structure, control structure element and control structure component.
• The friendly fire example (http://onto.fel.cvut.cz/ontologies/stamp-friendly-fire-example/current/d...) does not have any of the classes defined, e.g. ObjecAtRisk. Also, it is very odd to have an instance of a specific class also be in instance of a restriction. E.g. Two_F-15_fighter_aircraft is an instance of ‘anything that is a participate of a hazard’. It would make more sense to have an explicit triple saying what hazard it is a participant of.

The authors have addressed many of the reviewers' concerns, and improved the language, though there is still room for improvement of the English (a few errors are listed below, but there are many more). I also appreciate the explanation in the response of some of the ontological jargon, but please fold this in to the paper, so that it is more accessible; examples would help (eg. perdurants vs endurants).

However, my main problem with the paper remains its motivation. I just don't understand what problem this work is addressing. It's something to do with making connections between hazard analysis, safety management, STAMP, and ontologies, but is the goal to explain safety analysis to STAMP people? to find a case study for ontologies? to introduce ontologies to STAMP people? to provide a basis for functionality for STAMP tools? The abstract implies something much grander - a contribution to safety engineering, but this is not justified by the work.

abstract:

comparing to the traditional ones - can you give a citation for this (in the main body if not in the abstract)? which traditional SMS are you referring to?

I don't understand the connection between the evolution of SMS and the need to provide an ontological representation of STAMP

enable detection of safety-related problems and predicting them - there is nothing in the paper showing how this would be done

a STAMP hazard risk ontology that could help in analyzing accidents - if this is your primary motivation, it's not shown in the paper

we tested our ontology on realistic examples - tested what exactly?

sec 1

there has been a growing interest in building safer systems - citation? Are you really saying people were less interested in safe systems before some point?

were proposed over the years -> have been proposed ..

The two most recent systemic causation models and safety methods - papers are coming out all the time, you can't call these *the* two most recent

ad-hoc -> ad hoc

p2

One of the key obstacles of adopting these models in the industry is the lack of their formalization - citation please; some would argue that formalization has the opposite effect

this papers -> this paper

serve successfully -> have served ..

p3

It is clear that the theory of STAMP updates conventional definitions of hazard to include the system perspective (system state) - not true; it is usual in hazard analysis to consider system states irrespective of whether STAMP is being used

In terms of STAMP, this is a system state - you could just as well omit "In terms of STAMP"

how often flock of bird -> how often flocks of birds

p6

in a a

The purpose of the ontology is to allow for the representation of knowledge gained through a hazard analysis such as STPA - does this mean that STPA does not already represent this knowledge? and what do you do with it once you've represented it?

Def 4 - what is this a definition of?

We designed SHRO to describe safety issues and increase the awareness of safety models and methods in the industry - which industry? are you claiming that this ontology is needed to explain the concepts of hazard and risk to safety practitioners? or to STAMP researchers?

p11

Building safer systems requires putting emphasis on system hazards and eliminating or reducing their occurrence. - that's only half of what goes on; you can also mitigate them once they occur

p16

So if I understand Table 2, "ontology verification" concerns representing certain questions from the domain as queries on the ontology? More specifically (at least for Table 2), it means checking if there is an axiom that defines each of a range of concepts. This is a somewhat impoverished notion of verification, but if that's what it means in this context then so be it. The problem I have with this (apart from its triviality) is that this is surely going to be true by definition of your ontology.

p19

no validation of systemic perspective - what does that mean?