Review Comment:
This paper proposes LLM4Onto, a framework for automatically constructing domain ontologies from raw text. The framework extracts nouns and noun phrases, clusters them using affinity propagation, employs a large language model to assign ontology labels, and introduces the Head–Tail–Relation–Ontology, or HT-R-O, paradigm for deriving relationships between ontology classes.
The paper addresses a relevant problem and presents a potentially useful combination of conventional natural language processing, clustering, and large language models. However, the contribution is currently difficult to assess because its novelty is not positioned clearly, several central claims are insufficiently supported, important implementation details are missing, and the experimental evaluation does not adequately establish the method’s advantages over existing approaches.
Strengths
S1. The paper addresses an important problem.
Automatic domain ontology construction remains a relevant and difficult task, particularly because manual ontology engineering requires substantial time and domain expertise.
S2. The framework attempts to reduce dependence on large language models.
The combination of linguistic preprocessing, clustering, and selective use of an LLM may offer advantages in scalability, reproducibility, and cost compared with fully end-to-end LLM-based approaches.
S3. The paper considers both ontology concepts and inter-concept relationships.
The HT-R-O paradigm attempts to move beyond concept induction by also deriving relationships between ontology categories.
S4. The experiments cover several types of input data.
The use of dialogue, encyclopedic, academic, and cybersecurity data suggests an effort to test the framework under different domain conditions.
Weaknesses
W1. The originality and practical advantages of the contribution are not clearly established (see C1–C2).
W2. Several important claims in the introduction are unsupported, overly broad, or insufficiently defined (see C3–C5).
W3. The related-work discussion does not provide a systematic comparison with existing approaches (see C6–C7).
W4. The methodological description omits essential implementation and validation details (see C8–C12).
W5. The experimental evaluation is not sufficient to substantiate the paper’s main claims (see C13–C20).
W6. The provided materials do not appear sufficient for replication (see C21–C24 and the artifact assessment).
W7. The manuscript requires substantial proofreading and presentation improvements (see C25).
Detailed Comments
Originality and significance
C1. Clarify the central novelty.
The contribution does not currently stand out clearly from prior work on ontology learning, ontology generation with large language models, clustering-based schema induction, and relation extraction. The authors should state precisely which part is novel: the complete pipeline, the use of affinity propagation, the HT-R-O paradigm, or the specific combination of these components.
The statement that each component addresses limitations of existing text-based ontology-construction methods is too vague. I recommend adding a comparison table in the related-work section. The table should compare LLM4Onto with representative methods according to input type, supervision, predefined schema requirements, concept induction, taxonomy induction, non-taxonomic relation induction, human validation, LLM usage, and evaluation setting.
C2. Explain the practical significance and advantages over existing methods.
The paper would be stronger if it explicitly discussed whether LLM4Onto is cheaper, more reproducible, more scalable, or less dependent on prompting than end-to-end LLM approaches. Using deterministic preprocessing and clustering for part of the pipeline may reduce the number of LLM calls and improve reproducibility, but these potential advantages are not quantified or discussed in sufficient detail.
The authors should report computational cost, number of LLM calls, token usage, runtime, and sensitivity to model choice. These results would help establish the practical significance of the contribution.
Introduction and motivation
C3. Support the literature claims with citations.
The introduction states that some studies extract latent ontological patterns from structured data, whereas others use large language models to construct ontology frameworks. Representative citations should be attached directly to both parts of this statement.
The related claim that previous research has overlooked relationships between ontology categories is also too broad. Existing ontology-learning and ontology-engineering work has considered both taxonomic and non-taxonomic relationships. The authors should narrow the claim to the specific class of methods they intend to distinguish from LLM4Onto and support it through a systematic comparison.
C4. Define “intermediate-state data.”
This term is central to the paper’s motivation but is not clearly defined. It appears to refer to information that has already been processed into representations such as entities, triples, knowledge graphs, dialogue slots, taxonomies, or candidate concepts. The authors should define the term explicitly, provide concrete examples, and explain why beginning from such representations necessarily causes important semantic information to be lost.
C5. Avoid claiming that task decomposition inherently minimizes hallucination.
Breaking a complex task into smaller subtasks may reduce the model’s cognitive burden, but it does not by itself guarantee lower hallucination. Errors may arise during noun extraction, clustering, ontology naming, relation extraction, or relation normalization, and these errors may propagate through the pipeline.
The claim should therefore be softened or supported through a hallucination-specific evaluation, manual factuality assessment, or ablation study comparing direct ontology generation with the decomposed pipeline.
The “Technical Challenges” section would also read more effectively as discursive prose rather than as several long bullet points.
Related work
C6. Expand the discussion of traditional and pattern-based ontology construction.
The related-work section should discuss ontology-learning approaches based on lexical-syntactic patterns, Hearst-style patterns, statistical terminology extraction, ontology design patterns, and pattern-based extraction of non-taxonomic relations.
The current discussion of traditional ontology construction is too limited, particularly given that pattern-based ontology construction is directly relevant to the paper’s claim of constructing ontologies from text.
The authors should also discuss the relevant work by Lippolis et al. on ontology generation using large language models in more detail, rather than only including it in a list of citations.
C7. Improve the organization of the related-work section.
The current section combines ontology construction from scratch, ontology extension, ontology evaluation, ontology applications, and ontology-assisted systems. These are related but distinct tasks and should be organized into clearer categories.
In particular, work on supplementing or extending an existing ontology should not be presented as directly equivalent to constructing a domain ontology from raw text.
Method
C8. Specify which large language model was used.
The paper does not clearly identify the model or models used in the experiments. The authors should report the model name and version, provider, access date where relevant, temperature, top-p, decoding parameters, context-window size, prompt configuration, number of calls, and number of experimental runs.
Without this information, the experiments cannot be replicated.
C9. Cite spaCy and identify the pipeline used.
The manuscript should cite spaCy and specify the exact version and language pipeline used for part-of-speech tagging and noun-phrase extraction.
The authors should also describe the pronoun-filtering procedure and clarify whether the filtering lexicon was manually created, generated by an LLM, or constructed through another process.
C10. Report the embedding model.
The paper refers generally to “the BERT model,” but does not identify the checkpoint, pooling method, embedding dimensionality, similarity measure, or whether the model was fine-tuned for the target domains.
C11. Justify the removal of small clusters.
Discarding clusters containing fewer than (n) entities may remove rare but important domain concepts. The value of (n) should be reported, and the authors should analyze how this threshold affects ontology coverage, precision, and the treatment of infrequent domain-specific concepts.
C12. Explain the multi-round merging procedure.
The manuscript states that multiple dialogue rounds are used to merge semantically related clusters, but does not explain the stopping criterion, the exact prompts, how inconsistent model outputs are handled, or whether the procedure is deterministic.
Datasets
C13. Clarify what constitutes the “General Ontology” datasets.
The label “General Ontology data” is unclear. The authors should explain the origins of the Wikipedia and arXiv datasets, how the texts were selected, where the ground-truth ontologies came from, and whether these are established benchmarks or newly assembled datasets.
C14. Resolve inconsistencies in the dataset description.
The prose describes the Wikipedia ontology as four-level and the arXiv ontology as two-level, whereas Table 1 lists both as three-level. The reported numbers of nodes, edges, abstracts, and “abstract pairs” also require clearer explanation.
These inconsistencies make it difficult to understand the experimental setting.
C15. Explain the arXiv ontology.
It is unclear whether the arXiv ontology is based on official arXiv subject categories, a manually constructed ontology, or an ontology produced by another method.
The paper should provide an illustrative example showing an input abstract, the associated ground-truth classes, and the expected ontology edges.
C16. Clarify the status of the cybersecurity resource.
The cybersecurity material appears to be a text corpus rather than a dataset with a gold-standard ontology. The authors should distinguish clearly between the source corpus, the ontology generated from it, and any labels or reference structures used for evaluation.
If no gold ontology exists, this experiment should be described as a qualitative case study rather than a benchmark evaluation.
C17. Justify the benchmark selection.
The authors should explain why these datasets were selected and whether established ontology-learning or ontology-generation benchmarks were considered.
Without this discussion, it is difficult to determine how the experiments relate to existing literature and whether the selected datasets adequately test the paper’s central claims.
Evaluation and significance of results
C18. Strengthen the baseline comparisons.
The paper includes baselines for Wikipedia and arXiv. However, MultiWOZ is compared only with TeQoDO, and the cybersecurity experiment does not compare ontology-construction quality with a comparable ontology-learning method.
The authors should expand the baseline set and explain why each baseline is appropriate. Useful comparisons would include a direct text-to-ontology prompting baseline, a clustering-only baseline, an LLM-only baseline, and ablations without HT-R-O or multi-round merging.
C19. Do not compare ontology construction directly with NER label inventories.
Table 4 compares the number of ontology and relation types generated by LLM4Onto with the entity-label inventories of cybersecurity NER systems. These systems appear to solve a different task, and producing more types does not demonstrate that the resulting ontology is correct, coherent, or useful.
This comparison should be replaced or supplemented with expert evaluation and/or other metrics existing in the literature.
C20. Include a discussion and error analysis.
The evaluation section moves quickly from reporting scores to drawing conclusions. A separate discussion section is needed to interpret the findings more critically.
The paper should include examples of incorrect clusters, incorrect ontology names, omitted concepts, hallucinated concepts, incorrect hierarchy edges, and incorrectly normalized relationships. Errors should be traced to the component that produced them.
The relatively low Wikipedia results should also be discussed more cautiously. The claim that the method generates a more fine-grained ontology may be plausible, but it requires empirical evidence rather than being used as an explanation without validation.
Evaluation metrics and reporting
C21. Improve the description of the metrics.
Phrases such as “simple to calculate” are informal and do not explain the properties of a metric. The authors should instead describe what each metric measures, which types of errors it captures, and which limitations it has.
C22. Add qualitative examples alongside numerical results.
For each dataset, the paper should show a short input passage, the corresponding reference ontology fragment, the generated ontology fragment, and representative errors.
This is particularly important for the Wikipedia, arXiv, and cybersecurity experiments, whose data and ground truths are currently difficult to understand.
Appendix, materials, and reproducibility
C23. Expand the prompt documentation.
The appendix provides only the ontology-labeling prompt for the cybersecurity domain, although the method also relies on LLMs for cluster merging, relation extraction, ontology-level abstraction, and relation normalization.
All prompts, few-shot examples, model parameters, and prompt-selection procedures should be provided. The paper should also report whether the results are sensitive to changes in prompt wording.
C24. Provide code, configurations, and processed data.
No accessible implementation repository or complete replication package is provided in the materials available for review. The authors should release the code, prompts, preprocessing scripts, model configurations, etc.
Presentation and writing quality
C25. The manuscript requires thorough proofreading.
There are frequent grammatical errors, inconsistent capitalization, awkward expressions, and typographical errors in both the text and figures.
Examples include “an crucial,” “manual working,” “scrath,” “detailedly,” “Fristly,” “Reltaion extracted,” “arthitecture,” “Clutering,” and “Reltion.”
The word “cleverly” should be removed from the methodological description. Figure 2’s caption should begin with a capital letter and end with a full stop. Terminology such as “ontology,” “ontology class,” “concept,” “entity type,” and “taxonomy” should also be used consistently.
Assessment of the Data and Resource Artifact
A. Organization and README
Based on the materials available for review, I could not identify a complete, clearly organized replication package with a README that explains the directory structure, datasets, software requirements, execution steps, and expected outputs.
B. Completeness for replication
The provided resources do not appear complete for replication (e.g. no source code, etc.). The paper’s current methodological description is not sufficient to reconstruct these components reliably.
C. Repository suitability and long-term discoverability
No clearly accessible long-term repository could be identified from the materials available for review. Therefore, I cannot positively assess repository suitability or long-term discoverability.
D. Completeness of the data artifacts
The data artifacts are not sufficiently complete in their current form. The manuscript describes several datasets, but the relationship between the source texts, ground-truth ontologies, processed inputs, and generated outputs is unclear.
In particular:
the Wikipedia and arXiv resources are not documented clearly enough to determine how the ground-truth ontologies were constructed;
the cybersecurity corpus does not appear to include a gold-standard ontology;
dataset splits and preprocessing steps are not provided;
no examples of complete processed instances are included;
the generated ontology files used for evaluation are not available;
The authors should provide complete metadata and processing documentation even where the original datasets cannot legally be redistributed.
Overall Assessment
The paper addresses an important and practically relevant problem, and its hybrid design may have useful advantages in scalability, reproducibility, and cost. However, these advantages are not currently demonstrated convincingly.
The novelty is insufficiently differentiated from existing ontology-learning and LLM-based ontology-generation work. Central claims about hallucination reduction and gaps in prior research are overstated or unsupported. The evaluation does not adequately validate ontology quality across all datasets, and the discussion moves too quickly from numerical results to positive conclusions without sufficient analysis.
The lack of complete replication materials is also a major concern for a full research paper. The paper would require substantial revision to clarify its originality, strengthen its experimental comparisons, document the method and datasets fully, provide an error analysis and a proper discussion section, release materials for replication, and improve the overall writing quality.
|
