Eunomos, a legal document and knowledge management system for the web to provide relevant, reliable and up-to-date information on the Law

Submission in response to http://www.semantic-web-journal.net/blog/semantic-web-journal-special-is...

Solicited review by Miriam Fernandez:

The following paper presents Eunomos, a legal document and knowledge management system for the legal domain.

The introduction of the paper is clear and the topic, applying semantic technologies to overcome the current limitations of legal knowledge management systems, is definitely very interesting. The motivation and state of the art review highlight in detail the limitations of current systems and the gap that the authors are aiming to fill with this work.

My main concern is that no evaluation is conducted to assess neither the validity of the provided the results, nor the usefulness of the system in real use case scenarios. I am aware that the authors provide a description of several use cases/domains (section 5) in which the system can help users in their daily activities, but no assessment is done for the system.

Additionally, the approach is explained at a very high level. Although references to other publications are provided, the paper is not self-contained. More examples and further explanations of the different processes are required. For example, the authors mention that ontologies are "populated and updated manually" but that Triggers of PostgreSQL are needed to enforced consistency of ontology relations. Can authors please provide examples of such triggers and inconsistencies? Similarly, more extensive descriptions/examples are needed to explain how links are extracted from the documents, how paragraphs are interlinked, or how documents are classified (how many domains are used for the classification, what are "well populated domains"?, etc.).

The authors should also provide an estimation of the manual work needed (section 3.2.2) to ensure the reliability of the information provided by the system and explain what are the "human language technologies" used to facilitate the work of the knowledge engineer.

It is also important that authors justify the advantage of using semantic technologies in their approach. For what I understand in the paper, all the information is centralized in a unique relational database, including the ontologies. If the information of the ontology can be stored and managed by the DB, why use ontologies then?

To conclude, neither the documents, the ontologies, or a link to the system is provided. I would suggest the authors to provide links to the ontologies and a link or a video of the system, so that readers can have a better understanding of this work. Additionally, it will be also convenient to provide some statistics about the constructed ontologies (number of classes, relations, instances, etc.)

Solicited review by John Zeleznikow:

Why the acronym EUNOMOS?
I am underwhelmed by the Introduction: what is new and novel in EUNOMOS? Why should I spend my valuable time reading the paper?
Footnote 1: more details
Give Normattive website of Italian national legislation
At footnote 3 – US Supreme Court?
P4 – why not cite Susskind directly?
P5 – directly reference European XML sites
Mention what are weaknesses of ontologies in [23] and [35] and how EUNOMOS could improve upon them
P6 – too much emphasis on software, rather than explaining the important problems you are dealing with
Too much a system description rather than a research article!
The paper needs to be rewritten to focus upon research problems – what are the inadequacies of current systems, how does EUNOMOS address these inadequacies. Only then can you discuss the system – emphasising its novelty and how it addresses current limitations.
No need for so much code or diagrams – this should be a research paper rather than a system implementation

Solicited review by Adrian Paschke:

The authors describe a Semantic Web based document management system for legal documents, called Eunomos.

The paper is not a research paper, but a narrative system description of Eunomos and its features with a focus on the user interface.
Since the presentation is just a long narrative description which lacks any deeper details and thorough presentation and justification, a review of the research results is impossible. It is full of unjustified statements which would need to be proven and thoroughly evaluated. Since the paper touches on so many different aspects different research methodologies and evaluation techniques would apply from e.g. software engineering, information retrieval to user interface design and HCI, etc.

However, the most important step to improve the paper would be first to clearly define the research problems and then show the new contributions over the existing state of art. Just applying and implementing existing techniques to a specific domain does not make it a new research contribution. Among the many mentioned features and applied technologies in Eunomos DMS, I fail to see where there is a new original research. Many references are missing in the background section, e.g. to the described existing works on legal XML languages and ontologies. Also from the discussion of this related work it is not clear how the presented work goes beyond the state of art.

Overall, with 23 pages and double column formatting (which makes it more than 50 pages in single column format) the paper is quite long. The English needs to be improved. It should be written in a much more compact and concise way. The presentation of the important contributions needs to be significantly improved and more details need to be given, e.g. by formal representations of the improvements of existing algorithms, formal description of the applied semantics (expressiveness etc.), pseudo-code of applied algorithms/methods, system architecture diagrams, workflow models, user interface design etc. And all claimed statements / contributions need be justified and thoroughly evaluated with appropriate evaluations methods and compared against existing state of art. Moreover, if technologies / methods from other domains are applied to the "new" domain of legal knowledge representation it needs to be demonstrated that this leads to new research problems and new research solutions.

I would suggest to resubmit the Eunomos paper as an application report or report on tools/systems, as this short paper format would help to make the paper more compact and focused.