Influence of event representations on natural language querying interfaces to ontologies

Tracking #: 2823-4037

Rita Butkienė
Algirdas Šukys
Linas Ablonskis
Rimantas Butleris

Responsible editor: 
Elena Demidova

Submission type: 
Full Paper
Information in knowledge graphs is ordinarily accessed via queries in formal languages. Formal languages, however, present an obstacle for non-technical users. To make semantic search more convenient, it is desirable to enable use of natural language queries. One of the difficulties encountered in developing natural language querying systems is a mismatch between the way how users express their questions in a natural language and the way how a knowledge graph is structured. Such a mismatch is called semantic gap. One of the solutions for bridging the semantic gap is to use SBVR vocabularies for translation between the natural and formal languages. In this article, we investigate how alternative variants of event representation schemas affect properties of natural language querying interface: the size of ontology schema and vocabulary, the performance of querying and data import, repository size and query complexity.
Full PDF Version: 


Solicited Reviews:
Click to Expand/Collapse
Review #1
By Sherzod Hakimov submitted on 23/Aug/2021
Review Comment:

The paper presents an evaluation of ontology schema variants built using Semantics of Business Vocabulary and Rules (SBVR)vocabulary. The focus of the paper is on analyzing the effect of each artifact corresponding to three ontology variants in terms of query time, data insertion time. The applicability of the presented method is intended for applying to question answering systems on event-centric data.


Motivation: The paper presents an introduction into the question answering field on how the natural language questions are transformed into executable queries with use of vocabularies or query patterns. It is not clear what is the main motivation of the paper from Section 1 where the paper proposes using SBVR vocabulary to create different variants and evaluate the performance on query execution and data insertion times. It lacks the explicit mentions of the contributions of the presented paper.

Presented Idea: The main idea of the paper is to use SBVR vocabulary for translation of natural language questions into SPARQL queries. The question answering is utilized in answering questions about certain aspects of events such as “what did the talk about”, “what did the person confirm”, “what did the person emotionally confirm”, “what did the person talk about in a given year”, “what did the person emotionally confirm in a given year”. These competency questions are explained in Section 4. The application side of this presented method is to apply on querying Lithuanian news portal to find information - using the competency questions.

Paper weaknesses: The presented paper has multiple flaws with regard to novel contributions and description of the presented idea.

It is not explained clearly how the translation of natural language questions using SBVR works. The paper mentions this part as the motivation in the introduction but falls short in giving more details except the mention of the reference number [27].
The limited applicability of the event data for question answering. The competency questions (Section 4) are focused on only 5 aspects of the events, which should include that somebody should talk about something. However, there are other types of events that do not include somebody talking about a certain topic. The presented idea is limited only to a handful selected event types (talking event) and question types. It is not clear whether the SBVR vocabulary based translation engine is capable of handling different formulations of questions that ask the same information as the competency questions, e.g. “What was the person’s speech about?” <-> “What did the person talk about?”.
The main idea of the paper is defining 3 variants of the ontology schema to model the “talking event” and evaluate the performance of data insertion and query times (Section 5-6-7).
Small typos w.r.t. To table and figure references within the text. Almost all table and figure references are in small caps. The label Table 10 (Page 13) is left alone with the actual content. The resolution of the presented visualizations and data in tables and figures are low.
References: inconsistencies with regard to the information present in references. Some do not have publication years ([28]) or are written in different formats ([1] vs. [19]).

Overall: The novel contributions of the paper are limited based on the justifications provided above. The paper lacks in identifying a research question and providing possible solutions for it. The presented idea consists of defining three variants of ontology schemas and comparing them in terms of query and data insert times.

Review #2
Anonymous submitted on 01/Sep/2021
Major Revision
Review Comment:

This paper studied how different approaches to representing events affect different factors of natural language querying interface to ontologies, including SPARQL query execution time, data import time, number of triplets, and the size of ontology schema and vocabulary. After reading the article, I have the following concerns:

The structure and expression need to be highly improved. I suggest the authors write a related work section to summarize some related literature to improve the readability for non-expert readers. I also highly recommend the authors put the schema variants V1-V3 as a separate section to ease the readability. Even the content of the last paragraph in the introduction section for the paper structure is wrong. The section content is not consistent with what the authors write in the introduction section.

The main motivation of this work is missing. The authors should clearly state why they want to study how different approaches to representing events affect properties of natural language querying interface in the introduction section. What is the research gap there? Also, why do you select “talking”? why and how do you choose these 6 questions in table 10?

I also highly suggest the authors clearly summarize their contributions in the introduction section.

The authors show the differences in the explored factors SPARQL query execution time, data import time, number of triplets, and the size of ontology schema and vocabulary. However, the authors didn’t show if the differences are significant or not. The authors should do significant tests for them.

I suggest the authors write the limitations of this work and some future directions in the conclusion section.

The title of table 10 is on page 13 while the rest of table contents is on other pages, which looks a little bit strange. The title should be on page 11, not on page 10 independently.

Review #3
Anonymous submitted on 23/Sep/2021
Major Revision
Review Comment:

The paper concerns performance evaluation of a question answering system (data import time, query execution time, repository size depending on the chosen semantic representation of events). Topics of the paper match the Semantic Web Journal scope.

Unfortunately, it is hard to recommend the paper for publishing for the following reasons:
1. Absence of scientific novelty. The paper belongs to engineering papers, providing measurements and tests results, but the authors do not tackle any research problems within it. Significance of the results is not enough to be presented in the journal paper.
2. Vague content spread within the chosen paper structure. The paper is really hard to follow, too many details and too little clarity. In the introduction the authors discuss modeling problems in QA, like semantic gap, but the results (which are also unclear) are not connected to the methods of solving semantic gap. A lot of space is given to SBVR vocabulary, but is not clearly compared to any of existing event models (Simple Event Model, Event ontology, event modeling approaches in commonsense knowledge graphs). Description of SVBR and its content is split over the paper and the domain structure and statistics of that vocabulary are not disclosed.
3. Weak methodology. If I understood it correctly, the authors are focusing on evaluating, how different event representation models and OWL dialects influence the amount of generated metadata, execution time, etc. But those issues are not influencing the natural language understanding part of QA interfaces or semantic gap. It would be nice, if the authors clear the paper contributions and rewrite the corresponding parts accordingly. For now, the paper looks like several loosely coupled parts: intro touches upon the methods of QA interfaces, middle part describes a framework/vocabulary of patterns for semantic parsing/ event labeling, experiments come from measuring performance over different repositories. The actual takeaway for event modeling with Semantic Web technologies remains unclear.
4. Discourse model within SBVR is not compared or justified across existing discourse theory models. Since the authors want to model discourse (“talking”) as an event, why not consider Speech act theory, because communication is much wider that talking or talking emotionally and includes speech acts as “promising”, “asserting”, “commanding”, etc., see “Communication” model in the FrameNet as well and other Event representation models. What are the benefits of using the authors’ approach?
5. Complexity estimation (eq.1). Where do numbers 6 and 4 come from, are they related to the number of features in the model. Was it possible to use existing metrics for that task?
6. English needs proofreading.