Knowledge Engineering with Large Language Models: A Capability Assessment in Evaluating Ontology Property Restrictions

Tracking #: 4073-5287

Authors: 
Stefani Tsaneva
Guntur Budi Herwanto
Majlinda Llugiqi
Marta Sabou

Responsible editor: 
Guest Editors 2025 LLM GenAI KGs

Submission type: 
Full Paper
Abstract: 
Advancements in large language models (LLMs) offer opportunities for automating challenging and time-intensive Knowledge Engineering (KE) tasks. Constructing an ontology is a complex process, particularly when logical restrictions are modeled or when the development is performed by novice knowledge engineers or domain experts with limited training in KE. Consequently, developed ontologies often contain modeling errors, undermining the success of ontology-based applications and hindering subsequent KE tasks. Thus, it is important to investigate how LLMs can support KE tasks such as the evaluation of ontologies, which includes, among other tasks (e.g., inconsistency detection, competency question alignment, etc.) the detection and correction of errors in knowledge-based resources. However, challenges remain in systematically evaluating LLM performance and comparing different models in terms of their capabilities to perform concrete KE tasks. Moreover, there is a lack of comprehensive, task-specific benchmarks needed for such LLM capability assessments. As a result, selecting the right LLM to effectively support knowledge engineers presents a nontrivial problem. To fill these gaps, this study investigates how and to what extent LLMs can support four concrete (but not exhaustive) ontology evaluation sub-tasks: the detection, classification, explanation, and possible correction of modeling issues in OWL ontologies, focusing on the use of existential, universal, and cardinality property restrictions. To this end, we construct a benchmark dataset based on student-built ontologies and perform experimental assessments of the performance of four LLMs--GPT-4o, Claude Sonnet, DeepSeek V3, and Llama 3.3-- on these four KE sub-tasks. Additionally, we exemplify the definition of an annotation framework for the qualitative evaluation of LLM outputs and perform a comparative analysis of each model's capabilities. Our findings reveal notable differences in model behavior and task-specific strengths, underscoring the importance of selecting the most appropriate model for a concrete KE task.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 01/May/2026
Suggestion:
Accept
Review Comment:

The authors have carefully revised the manuscript in response to the reviewers comments, and it is now suitable for publication.

More specifically:
- The framing of the research questions has been improved for clarity.
- The dataset availability section has been strengthened. I appreciate the care taken to address data leakage concerns, as explained in the rebuttal regarding why the dataset is not shared.
- The experimental setup has been clarified by adding explicit temperature settings for each experiment.
- Appendix B, containing the ablation study, has been added.
- And many other minor changes that I noticed.

I would like to thank the authors for their thorough revisions and the quality of this work. In its current form, the manuscript is ready for publication.

Review #2
By Bohui Zhang submitted on 04/Jun/2026
Suggestion:
Accept
Review Comment:

This is a review of a revision of a previous submission, for which I had already recommended acceptance. In the revised version, the authors have addressed the recommendations and concerns raised by all reviewers, including my own. I therefore continue to recommend acceptance.

Review #3
Anonymous submitted on 15/Jun/2026
Suggestion:
Minor Revision
Review Comment:

I want to express my thanks to authors again for addressing the comments and the substantial effort for improving the paper, especially doing extra ablation study. This version has improved the clarity and consistency of the paper for a large part.

The authors are able to address most of the concerns mentioned in the last round of review. Well done! However, two minor concerns remain, which I believe could addressed to further improve the clarity and fidelity of the paper.

Firstly, on section 3.2 Task 2, I thank the authors' added explanation on classification problem, however, on the Evaluation section, the paper says: 'Task 2 can be assessed using standard multi-label classification metrics, including weighted accuracy, precision, recall, F1-score and hamming loss'. The authors may also wish to explicitly mention the evaluation metrics for single-label classification problem? The evalution metrics between single-label classification problem and mutiple label classification problem may overlap, but they are not exactly the same.

Secondly, this concern relate to R3C7, R3C8, R3C9, R3C10, and R3C11, I want to thank the author for detailed explanation and ablation study. The paper uses (constraint, restriction, property restriction) interchangeably creating a wording mismatch between the prompt and main part of manuscript. They authors provided a mapping table on Appendix A2, aligning the terms used in prompt and terms and types used in the paper writing. To avoid confusion and ensure consistency, the authors may want to justify why the paper use (constraint, restriction, property restriction) interchangeably and consider moving the mapping table and the explanation the main text rather than just in appendix. This would not only help reducing the need for readers to cross-reference Appendix A.2 while reading the main text but also avoid confusing if reader were not crossing referencing to appendix. At the current state, this detail can be easily overlooked by readers.