Extensive Benchmark of Frugal Encoder-Decoder Language Models for Datatype Properties Extraction and RDF Knowledge Graph Generation

Submitted by Célian Ringwald on 11/06/2025 - 02:19

Tracking #: 3971-5185

Authors: 
Célian Ringwald
Fabien Gandon
Catherine Faron
Franck Michel
Hanna Abi Akl

Responsible editor: 
Guest Editors 2025 LLM GenAI KGs

Submission type: 
Full Paper
Abstract: 
The choice made for representing the inputs and outputs of generative pre-trained language models (PLMs) can impact their fine-tuning on a new task. This article focuses on the fine-tuning and linearization process to generate facts extracted from text. On a restricted relation extraction (RE) task, we challenged five encoder-decoder models including BART, T5, CodeT5, FlanT5 and PileT5 by fine-tuning them on 13 linearization variations, including RDF standard syntaxes and variations thereof. Our benchmark covers the validity of the produced triples, the model's performance, the training behaviour and the resources needed. We show these PLMs can learn some syntaxes more easily than others, and we identify a promising ``Turtle Light'' syntax supporting the quick and robust learning of the RE task.
Full PDF Version: 
PDF icon swj3971.pdf
Previous Version: 
Extensive Benchmark of Small Language Models for Datatype Properties Extraction and RDF Knowledge Graph Generation
Tags: 
Reviewed
Long-term Stable Link to Resources: 
https://github.com/datalogism/12ShadesOfRDFSyntax

Decision/Status: 
Accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 16/Mar/2026
Suggestion:
Accept
Review Comment:

I thank the authors for their revised work, which addressed the concerns raised in the previous revision round. The current version provides a better, streamlined report across all sections and clearly distinguishes itself from their prior work. The paper provides a valuable contribution to the research community in the form of an evaluation of several small language models (SLMs), specifically encoder-decoder models, on relation extraction tasks. Furthermore, it provides a detailed analysis of several characteristics that may affect the performance of SLMs, including (a) RDF serialization, (b) additional constraints imposed on the language models, (c) triple ordering, and (d) the use of explicit instructions. In addition, the datasets and source code for the evaluation are available and can be (re-)used for further research addressing similar challenges.

The feedback for the current version of the paper is about the paper's structuring and presentation, as follows:
* Introduction: inconsistent indentation with paragraphs and research questions.
* Fig.4.: listing instead of figure
* Table 4: consider adding the code for each serialization for user reminder (e.g., vt1f for factorized one-liner turtle light)
* Section 4.2: consider adding sub-sections to help with readability
* The authors can position the figures and tables (e.g., Fig.14) closer to the corresponding text explanation.

Review #2
By Pablo Calleja submitted on 17/Mar/2026
Suggestion:
Accept
Review Comment:

The authors have adequately addressed the various points raised in the previous round of review.