Coupling conceptual modeling and rules for the annotation of dramatic media

Review for paper: “Coupling conceptual modelling and rules for the annotation of dramatic media”

The paper proposes a representation combined with reasoning rules over dramatic plays, allowing for reasoning over plans, actions and emotions. Representation is based on OWL and SWRL rules and the applicability of the approach is demonstrated by using it on a specific scene from Shakespeare’s Hamlet. Specifically the evaluation section consists of the application of the proposed representation and rules for analysing different versions of the same scene. The paper is well written, containing very few typos, and provides an extensive and detailed description of the related topics and background. The rationale behind design decisions is well explained and these decisions are based on related work on analysis of drama.

The evaluation does not provide an indication of the required time for annotating a play using semi-automatic methods. Providing such information is useful since the applicability of the proposed work depends also on the time required to annotate data. On the other hand the ontology is independent from the annotation mechanism, thus the ontology is useful even if the annotation process is currently time consuming. Also custom rules are used based on existing literature on the topic of drama, which seems sound for the specific case that is analysed. But the results obtained by applying these rules (e.g., number of conflicts in Table 1 of page 20, or sequence of emotions in Table 2 of page 20) should be compared to the results obtained by human experts in order to estimate the precision and recall of the proposed representation and SWRL rules. In addition to that, a more rigorous evaluation should contain examples of different plays, modern and/or ancient in order to demonstrate the overall soundness and applicability of the proposed representation. For example a set of rules that is efficient for analysing a Shakespeare’s Hamlet may not be efficient (in terms of precision/recall when compared with analysis results from human experts) for analysing a Hollywood movie or an ancient Greek drama.

Summarizing: the paper is well written and the proposed representation seems sound and useful. Providing information about the time required to annotate a drama or scene would be desirable. Comparing results with analysis results of human experts in terms of precision and recall is necessary. Using the ontology and reasoning rules for analysing additional dramas (preferably from different time periods) and evaluating the results with respect to human expert analysis results is highly desirable.

In this paper the authors describe a semantic framework for representing and reasoning over conceptual and emotional features of drama. Overall, the article is well written and technically exhaustive, although in my opinion minor revisions are required. In the following, I will propose some improvements from the viewpoint of general argumentations, specific problems, and organizational issues.

p.3, section 2.2: The notion of “Agent” here should be fleshed out with more details, for instance referring to literature in Cognitive Science. I’d specifically recommend looking into “bounded rationality” as defined by Herbert Simon [1], which the definition of an agent as “a willing dynamic entity that constantly appraises the state of the world by means of rational deliberation…” seems to originate from.

p.7, figure 1: the part on the top left corner is barely readable, please enlarge it.
Section 5: 1) the authors should introduce all the OWL 2 sub-languages, motivating why they end up picking OWL 2 RL for the implementation of the Drama Ontology. As a matter of fact, this choice is justified later in the paper by the need of using an ontology-driven rule based system for computing emotional and motivation features of drama, but Section 5 should at least contain some general considerations on the computational aspects of the OWL-RL in comparison with EL and QL; 2) the distinction between Endurant and Perdurant is principally described in [2] (see below), although referring to [38] is also relevant.

p.10, figure 3: the figure includes a diamond representing “VerbNet”, but the resource has been introduced neither on the text nor in the caption. Please either remove it from the figure or say something about VerbNet, e.g. in terms of the differences with FrameNet.

pp. 11-12: I’d suggest to reduce Section 6, focusing more on recent developments of the integration between ontologies and rule systems rather than lingering on general aspects like horn clauses.

Section 6.1: The approach of mapping OCC theory to a set of ontology-grounded rules is original and supported by thorough examples in this section. Nevertheless, it’d be interesting to compare it with a similarly comprehensive system presented by Clark Elliot in 1992, the “Affective Reasoner” [3].

Section 7: This section doesn’t seem convincing and it’s probably the weakest part of the article. Of course I understand that this work has been completed only recently and that a proper evaluation is planned as future work. But no clear methodology and metrics are provided for testing the functionalities of the system. Moreover, Table 1 and 2 contain just a qualitative assessment in terms of “units” and “conflicts” identified by a single student. The authors should expand this section, mentioning how they intend to create and consolidate a suitable evaluation dataset, e.g. exploiting Amazon Mechanical Turk for producing "ground truth"

My final recommendation to the Editor is to accept the paper with minor revisions.

Typos:

p.11, second column, third paragraph “[…] a W3C proposa, l that, “. To be turned into “a W3C proposal, that”
p.12, second column, line 6: “sermantics” / semantics

References:

[1] H. Simon (1991) “Bounded Rationality and Organizational Learning. Organization Science , 2 (1), 125-134.

[2] Borgo, C Masolo (2009) Foundational choices in DOLCES Handbook on ontologies, 361-381.

[3] http://condor.depaul.edu/elliott/ar/papers/dis/elliott-phd.html

The paper describes a semantic system for the representation of drama features and reasoning about them. The aim of this system is to apply ontological representation and reasoning to formalize the dramatic qualities of media objects. The contribution of the paper is described in two parts: an ontological model and a set of rules that define the reasoning layer.

An important shortcoming of the paper is that it is extremely unclear as to what is meant by "drama". Section 1 takes it as given, and just describes its growing importance relative to the digital world. Section 2 reviews existing work on the subject by enumerating a number of possible definitions and listing a number of "dramatic elements". But the definitions provided do not coalesce into a sufficiently defined single vision, and it is unclear whether the described dramatic elements are intended as a set of constituents of a representation of drama or merely as selected features that have a potential use in the annotation of media objects with dramatic qualities. This is important, as the dramatic elements are later encoded in the Drammar ontology, which is employed in an annotation tool suited that is the core contribution of the paper. The authors should clarify whether they consider this set of dramatic elements a necessary and sufficient representation for drama, or just for the annotation of media objects, or whether they simply have selected these elements from a larger set that might also have been used. This is important because it determines whether the ontology is proposed as a closed set of classes for this domain, or whether it can be extended in the future with additional elements at need.

Section 5 provides a description of the Drammar ontology. It is a non-exhaustive description of some its constituents. This is acceptable. However, care should be taken to ensure that at least those elements of the ontology required to understand the rest of the paper are described. As it stands, no definition is given of elements Role and Schema, which play an important part later in the paper (page 13, description of the mapping rules). In contrast, a lot of space is devoted to the element ExternalReference, which does not seem terribly relevant to the rest of the paper. An example of annotation is provided. This has suffered considerably in transcription, with the labelling of the various elements being inconsistent across the figures, the caption of the figures, and the text where they are paraphrased. An effort should be made to correct this. Even if the labels matched, there are two main considerations to be made here. One concerns the limitations of the format for explaining the complexity of the subject. The ontology can only be partially described within the space available in the paper. Only a subset of the annotated example seems to be shown in the figure. For a reader unfamiliar with the ontology and the notational conventions, the task of understanding the figures and following the arguments provided along with them becomes titanic. Unless the authors find a better way of explaining the information being represented and how the representational mechanisms operate, their efforts at communicating the value of their contribution run a risk of being unsuccesful. If drama scholars are to be convinced that the proposed notation is a worthy addition to their current toolset, a worked out example that remains cryptic after being presented in several diagrams and 23 pages of text is not a strong argument. The difficulty in understanding an annotated example is far greater than your run of the mill drama scholar is ever likely to contemplate as a communication medium, let alone as a language to actually use in his everyday work.

Section 6 describes the reasoning layer. This section is very badly written. It mixed up description of technologies employed with the paper's technological contribution (see details on this below). It mashes together high-level description of the functionality with low level details of the technologies employed. It fails to deliver a clear description of its functionality. the English is at times particularly cryptic (for instance, the first paragraph of section 6.1 is particularly difficult).

Part of the problem is that it talks about "the latter phase" and "the second step", probably referring to the mapping phase and the appraisal phase, which were mentioned in passing at the end of the previous section. This concepts seem to be reasonably important for the system. They should be described in detail at an acceptably high level of abstraction so that the reader has a reasonable idea of what the purpose of all the technical machinery described later is. If these phases corespond to what is being described in subsections 6.1 Mapping rules and 6.2 Emotion and SWRL Rules, this should be explained more carefully, and consistent labelling should be applied to help the reader establish the relationship between different parts of the paper.

Part of the problem in this section is that it is unclear what the input is to which the rules are applied. A timeline is mentioned, but no example is given of what one looks like. The fact that the example given seems to refer to parts of the representation that are not covered by the excerpt presented in Fig. 4 makes it even more difficult to understand. For the procedure to be understood, it is imperative that examples of the timelines involved be provided.
Fig. 6 is also difficult to understand. To improve it, it might be useful to include in the caption references to the labels given to elements in the picture, so that the reader can more easily match concepts described in the text with blocks in the picture.

The process of ascertaining emotions from the representation is very interesting, and a working solution would be very valuable. The description of how the rules operate is also a little difficult to understand.

An important objection arises here. The rules seem to be hand-crafted based on the author's intuitions about the particular emotional labels they are employing. Two different approaches are possible here. One is that the authors consider that each user of the system should hand code the rules that define the effect of emotions for his desired domain. Another is that the authors consider that the rules based on their own intuition have universal applicability across domains, dramatic situations, and media objects. The first approach again comes up with the difficulty of each drama scholar having to hand code the rules he needs, and the lack of universal applicabilty that would result thereof. The second approach stretches the confidence that prospective users have to deposit in the system. If the assumption underlying this second approach were to be considered, the set of rules would have to be validated against a large set of examples, over which judgements by human evaluators would have to be collected, and comparisons and rates of success would have to be considered.

In general terms, if the experience of 60 years of knowledge representation have shown us anything it is that a choice of representation for a particular subject constrains the set of operations that can be carried out over it. The underlying assumption of the whole paper is flawed in the following sense. The set of concepts chosen for inclusion in the representational ontology will only include a subset of all the possible ones that might have been contemplated for any given dramatic situation. As a result, a number of possibilities for that particular situation will be lost when it is represented using this ontology. Similarly, the set of possible emotional reactions considered by the rules provided will be only a subset of the possible emotional reactions in any given situation. Again, a set of possibilities for the dramatic situation will have been lost. This time the error will be larger as it will compound errors at each level of representation, which build upon one another. As a result, the representation that may be obtained in this way will be at most a crude approximation to the simplest possible analysis that could be made of the situation. Drama scholars tend to devote enormous amount of text to describe even the shortest scene, and to consider several possible interpretations, possibly in opposition of one another. They are very unlikely to spring at the chance of having the task oversimplified by a representational mechanism that is difficult to understand and that restricts their options to a limited subset of the ones they can think of, and which does not allow for nuance in the expression of subtle differences.

Having said that, I think the technical contribution of the paper is valuable. My objections concern more the claims on its usefulness and its potential for impact on the world of drama scholarship as described in the paper.

The contribution is ambitious and it has substantial potential to become valuable. But it has two major failings.
One concerns the inflated claims for the imperative need and expected usefulness of the kind of representation described. These claims should either be argued more carefully or toned down.
Another has to do with the way the material is presented. as it stands it would require severe rewriting to reach a standard where it can be perceived as coherent and convincing by readers of the journal. It should be substantially improved.
These two problems are, of course, somewhat related, as discussed in the text of the review above.

Overall the paper has a slight problem relative to structure. It is traditional in scientific papers to separate the material that is considered an original contribution of the authors from the material that existed before and is just described because the contribution relies on it. This rule is violated repeatedly in this paper. Section 5 includes a description of BDI theory of agents (page 8, 2nd col) and the SUMO and YAGO ontologies (page 9, 1st col). Section 6 includes a revision of possible strategies for integrating ontologies and rules (page 11, 2nd col and page 12, 1st col) and a brief review of models of emotion (sprinkled over pages 14 and 15). In this case, it becomes particularly difficult to spot where the authors' contribution starts. An effort should be made to clearly separate what is description of prior work being used as foundation here, and the author's own contribution.

A number of minor edits are needed:

- page 3, 1st col, 2n par, line 11: "non dramatic" -> "not dramatic"
- page 5, 1st col, 3 lines before the end of 2nd par: "extention" -> "extension"
- page 5, 2nd col, 2 lines from beginning: citation missing
- page 6, 2nd col, last par, 2nd line: "mapping model" -> "mapping module"
- page 8, 2ns col, 4 lines from end: "who" -> "which"
- page 12, 2nd col, 6 lines down the page "underlyng sermantics" -> "underlying semantics"
- page 12, 2nd col, halfway down the page "mechanismsms" -> "mechanisms"
- page 12, 2nd col, 12 lines from bottom "possibile" -> "possible"
- page 12, 2nd col, 5 lines from end: "figure Fig. 5 synthetizes" -> "Fig. 5 synthesizes"
- page 12, 2nd col, 3 lines from end: "connectig" -> "connecting"
- page 12, 2nd col, 2 lines from end: "appraisal one" -> "appraisal function"
- page 19, 1st col, 8 lines from end, "effects they hold" -> "effects that hold"
- page 19, 2nd col, 14 lines into 2nd par, "succeeds in calculate" -> "doesn't mean"
- page 19, 2nd col, 20 lines into 2nd par, "doesn't means" -> "succeeds in calculating"
- page 19, 2nd col, 3rd par, "basilar feature" not defined
- page 19, footnote 4 "stage direction" -> "stage directions."
- page 20, 1st col, 7 lines into 1st par: "an tightened pace" -> "a tightened pace"