A Conceptual Model for Detecting Interactions among Medical Recommendations in Clinical Guidelines

Tracking #: 739-1949

Authors: 
Veruska Carretta Zamborlini
Rinke Hoekstra
Marcos da Silveira
Cedric Pruski
Annette ten Teije
Frank van Harmelen

Responsible editor: 
Guest Editors EKAW 2014 Schlobach Janowicz

Submission type: 
Conference Style
Abstract: 
Representation of clinical knowledge is still an open research topic. In particular, classical languages designed for representing clinical guidelines, which were meant for producing diagnostic and treatment plans, present limitations such as for re-using, combining, and reasoning over existing knowledge. In this paper, we address such limitations by proposing an extension of the TMR conceptual model to represent clinical guidelines that allows re-using and combining knowledge from several guidelines to be applied to patients with multimorbidities. We provide means to (semi)automatically detect interactions among recommendations that require some attention from experts, such as recommending more than once the same drug. We evaluate the model by applying it to a realistic case study involving 3 diseases (Osteoarthritis, Hypertension and Diabetes) and compare the results with two other existing methods.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
[EKAW] combined track accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 16/Aug/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

2

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

3

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

3

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

2

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Review
Please provide your textual review here.

I think this paper is worth publishing as conference paper, but not yet as a journal paper. It takes a previous model for a clinical guidline and extends it to deal with the problem of of interactions between guidelines for co-morbidities. The model has not been implemented and the only evaluation is a discussion of how it might apply to a simple case study. Although the extension to the previous model is novel and interesting, the authors comment that it needs further extensions e.g. to deal with temporality and hierarchies and to be applied more detailed case studies. I think a journal publication would be more appropriate at that stage.

The EKAW call for combined conference/journal papers expects papers to be significantly extended for the journal. The areas to be extended for the journal paper have normally been reported in summary in the conference paper. The extensions suggested in the paper seem rather to be future work, but the authors may wish to clarify this with the editors.

-------------------------------------------------------------------------------------------
I now want to make some more general comments about this type of system, separate from my review comments above. Although the paper is a perfectly reasonable piece of research, it is a very, very long way from being of clinical use. This is not because it hasn’t been implemented and deployed, but because it doesn’t address the open-ended nature of the clinical situation.

The paper gives the example of a drug being recommended by one guideline but contra-indicated according to another guidline. This is fairly clear-cut, but in practice two drugs may not be contra-indicated but if X is recommended by one guidline, then Z recommended by another guidline might be preferred to Y when given together with X. This might be for medical reasons, but might be for reasons as simple as palatability, timing of doses so drugs can be taken together etc etc. For effective treatment it can be critical to get this right. Palliative care is probably a good example. The precise cocktail of drugs that will best manage a patient’s pain depends on many many factors beyond contraindications and it can be very distressing for the patient as well as staff and visitors if the combination is not right for that particular patient. In fact one can imagine, that not only with interactions between drugs, but that virtually every recommendation from a clinical guidline, might change slightly given a recommendation about a co-morbidity. One might respond that clinical guidelines should be evidence-based and should not take into account such minor issues. I don’t think this is realistic, at the end of the day the clinician will expect / hope that an automated advice system to give them precise advice.

I think it is therefore a little unfair of the paper to comment that previous authors “assume that all potential inconsistencies are manually detected by domain experts and rules are created to deal with them, often introducing new recommendations to address the conflict” Certainly it might be useful to automatically detect some well-defined inconsistencies, but since the possible ways in which recommendations might interact and need to be modified, is unknown every combination of recommendations will need to be considered. Again one might say the problem will be solved with more comprehensive guidelines and related ontologies. But is this in any way realistic?

I should explain why I see this as such a problem. There are few detailed studies on the maintenance of knowledge based system, but the anecdotal and other reports that do exist all suggest that the real challenge is not starting out, but finally adding in all the knowledge the system needs. Zacharias in a survey of developers identifies debugging knowledge as the major challenge – that is getting into the system all the knowledge it finally needs and getting it to work properly together [1]. Apologies for a number of citations to my own work in the following, but it provides a useful example. An early medical system that I had the good fortune to work on with was put into use when it was 96% accurate according to evaluation on test cases. It was maintained and adding the extra knowledge the experts gradually decided it needed took it to 99.7% accurate, but the knowledge base almost doubled in size in the process [2]. And of course it was increasingly irritating and difficult to make changes. This led to Ripple-Down Rules (RDR) which aimed to make it is a simple as possible to keep on adding new knowledge and refining existing knowledge. The RDR approach is not important here except that use of RDR provides some data on how people keep adding knowledge to a large number of medical expert systems over years [3]. [4] shows only one example, but in more detail showing that rules are still being added 8 years later. Of course this is due to RDR where rules are only added when a case needing a new rule or a correction occurs. However, in the 8 years the system processed 7 million patient reports. That is, even for a fairly simple domain like general biochemistry you need to see a lot of cases before you cover every pattern where a domain expert wants to recommend something different. With RDR domain experts are very unconstrained so of course they might be chasing trivial refinements that really don’t matter clinically, and that the move to clinical guidelines is to help constrain clinicians to more evidence-based decision making, but the data still suggests complexity and open-endedness in medical knowledge.

If we assume that there may be many possible, not yet identified interactions between multiple guidlines, then the critical question becomes how to deal with these. Clearly one cannot look at every combination of recommendations from guidelines, as huge numbers of these may never occur. However, it might be possible to identify when a new combination of recommendations is proposed with the clinician being warned this is a novel combination of recommendations and the combination being referred to a guidline authority. Again because RDR deals with cases as they occur, there has been some research on identifying novel cases e.g. [5] including methods more specifically focussing on novel combinations of conclusions [6]. Perhaps the problem here could be handled fairly simply

I am not suggesting for a moment that the authors should start applying an RDR approach to their problem, as this area certainly needs the type of model-development research they are undertaking. But having been invited to review this paper, it was an opportunity to emphasise that getting such a system into successful routine use resulting in satisfied users, there are major knowledge-related issues beyond the practical issues of implementation and deployment (itself a significant challenge) of such a model.

1. Zacharias, V.: Development and Verification of Rule Based Systems—A Survey of Developers. In: Bassiliades, N., Governatori, G., Paschke, A. (eds.) RuleML 2008, Orlando, Jan 1 2008, pp. 6-16. Springer-Verlag
2. Compton, P., Horn, R., Quinlan, R., Lazarus, L.: Maintaining an expert system. In: Quinlan, J.R. (ed.) Applications of Expert Systems, vol. 2. pp. 366-385. Addison Wesley, London (1989)
3. Compton, P., Peters, L., Lavers, T., Kim, Y.-S.: Experience with long-term knowledge acquisition. Paper presented at the Proceedings of the sixth International Conference on Knowledge Capture, KCAP 2011, Banff, Alberta, Canada, ACM: 49-56. (There was error in one section of this paper as outlined in comment section of the ACM Digital Library webpage, including a link to a corrected version, but this error does not relate to the discussion here)
4. Compton, P.: Situated cognition and knowledge acquisition research. International Journal of Human-Computer Studies 71, 184-190 (2013). doi:10.1016/j.ijhcs.2012.10.002
5. Finlayson, A., Compton, P.: Run-time validation of knowledge-based systems. In: Proceedings of the seventh international conference on Knowledge capture 2013, pp. 25-32. ACM
6. Dazeley, R., Park, S.S., Kang, B.H.: Online knowledge validation with prudence analysis in a document management application. Expert Systems With Applications 38(9), 10959-10965 (2011).

Review #2
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 1 weak accept

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 3 fair

Novelty
Select your choice from the options below and write its number below.

== 4 good

Technical quality
Select your choice from the options below and write its number below.

== 3 fair

Evaluation
Select your choice from the options below and write its number below.

== 2 poor

Clarity and presentation
Select your choice from the options below and write its number below.

== 3 fair

Review
Minor revision
This is well-written paper and mostly well organized. The aim of this paper is to define a method that allows evaluating set of recommendations and deriving certain types of interactions requiring little or no human intervention.

But I'm not convinced that the evaluation part meets the requirements for evaluation both from quantitative and qualitative point of views. I'm somewhat concerned that evaluation part in this paper is more framework of evaluation than show the result of evaluation to convince the future reader of this manuscript to use this method.
Having said that, I am prepared to recommend that the paper be accepted subject to revisions to address the points of concern listed below.
-Saying “It improves the reasoning capabilities for deriving interactions among recommendations within several CIGs. Although there is space for improvements in the current model, we believe the benefits from a more detailed semantics for the CIG elements can already be observed”. This is kind of general speaking and I suggest that when you talk about improvements you need to come with more robust results to show the improvement!
-Since this approach has not implemented yet how could you claim the below statements? “(i)(semi)automatable identification of inter- actions among recommendations; (ii) detecting interactions among several recommendations within several CIGs (instead of only pairwise combinations); (iii) (semi)automatable verification of the resultant CIG containing new recommendations eventually introduced to address conflicts”. Based on my understanding this method has not been implemented yet. Therefore, it is too strong to claim all above here. It could be completely ok if this paper just suggesting the framework, in this case you need to be more careful about what you claim and maybe need to rewrite those parts or implement the method!
-Another big concern is that you build up this model based on TMR model, which has not been published yet! (Zamborlini, V., da Silveira, M., Pruski, C., ten Teije, A., van Harmelen, F.: To- wards a Conceptual Model for Enhancing Reasoning about Clinical Guidelines: A case-study on Comorbidity. In: Knowledge Representation for Health-Care. Lecture Notes in Computer Science, Springer Berlin Heidelberg, Vienna, Austria (2014 - Forthcoming). It is hard of hard to accept and follow something that you refer to it without have that resource available. It would be ok if you refer and make something based on published work. Then reader and reviewer could follow your work! Maybe it was more reasonable that you waited to publish that work before submit the new work based on that.

- In this paper you claim to address the limitation such as re-using, combining, and reasoning over existing knowledge by proposing an extension of the TMR conceptual model to represent clinical guidelines that allows re-using and combining knowledge from several guidelines to be applied to patients with multimorbidities. My suggestion is in your new version try to address each limitation more clearly with individual your examples.

Review #3
Anonymous submitted on 02/Sep/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.
2

Reviewer's confidence
Select your choice from the options below and write its number below.
2

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.
4

Novelty
Select your choice from the options below and write its number below.
4

Technical quality
Select your choice from the options below and write its number below.
4

Evaluation
Select your choice from the options below and write its number below.
3

Clarity and presentation
Select your choice from the options below and write its number below.
4

Review
Please provide your textual review here.

The paper presents an extension of the TMR conceptual model, TMR4I, which addresses detection of interactions among medical recommendations in clinical guidelines by refining the recommendation concept in the initial model. The TMR4I model is evaluated in a realistic multimorbidity case study and its advantages are highlighted in comparison to two existing approaches. While the two approaches detect inconsistencies between pairs of clinical guidelines manually, TMR4I provides (semi)automatic detection within several clinical guidelines.

I would like to see answers to the following questions:
What does the TMR acronym stand for?
What are the disadvantages and the limitations of the TMR4I model?