From Scientific Variables to Knowledge Graphs: The I-ADOPT Benchmark

The paper identifies a specific need: a lack of corpora focused on the representation of scientific variables and a lack of benchmarks designed to evaluate semantic decomposition rather than simple triple generation (aka relation extraction or open information extraction). In provides a solution through the I-ADOPT framework and benchmark. This being said, the paper is the natural continuation of a set of papers and W3Recs published during the last 3-4 years.

The resource itself is new. The benchmark includes an expert-annotated corpus of 102 scientific variables spanning domains from Physical Sciences to Social Sciences. Unlike Automated Knowledge Graph Construction (AKGC) tasks, the focus is on the fine-grained semantic decomposition of variables (identifying properties, objects of interest, matrices, and constraints), a specialized task requiring domain expertise.

Since it establishes a new benchmark, the paper does not compare with previous benchmarks. This makes it difficult to fully evaluate the contribution. However, the results reveal significant insights into the current LLMs, namely
• i) Limitations - Even the best-performing large models (like Qwen-32B) struggle, achieving an F1 score of less than 50% (specifically 0.45).
• ii) Diminishing Returns on Scale - increasing model size has a limited impact, with only a 4% performance gain between 8B and 32B models. This particular results is supported by recent papers on smaller models like Transformers, and it was well known for a number of years, ever since the Vampyre architecture was published back in the late 10s.
• iii) Entity Linking Baseline: The study establishes a baseline for linking extracted values to Wikidata, showing that a reranking approach can achieve roughly 76% accuracy, a ten-percentage-point improvement over naive methods.
The latest contribution is particularly important, as there are many EL baselines, but most of them are focused on classic TAC-KBP types, and less on scientific entities. This being said, many of the modern NEL scientific benchmarks are not even mentioned in the SOTA. One such example is the STEM-ECR dataset (see https://arxiv.org/abs/2003.01006).

The paper is well-written, and well-structured. One minor thing that is missing would be an illustration that explains why this benchmark is useful, as such an image would definitely improve readability and make it more attractive to new readers.

In terms of FAIR, the paper does provide a clear breakdown of the I-ADOPT framework, the systematic design process for the corpus, and the specific prompts used for the LLMs. It also provides a critical self-analysis, that is much needed in times like ours. The authors are transparent about the difficulty of the task, noting that 46% of variables required interpretation by experts because definitions often contain implicit knowledge. The issue of implicit knowledge is a difficult one to tackle, even in pyschology where Nosek's work provided the framework for it for the past decades. However, in other domains, it rarely surfaces, therefore it is good that the authors recognize this issue.

The structured evaluation is generally okay. The use of both automated metrics (Precision, Recall, F1) and expert semantic analysis (grading outputs as Good, Acceptable, Weak, or Wrong) provides a comprehensive view of model performance that goes beyond mere numbers. However, in terms of metrics, it would be good to provide both accuracy and the fine-grained F1 (macro and micro) metrics. The derivation of exact and close metrics also deserves a picture. While me as a NER/NEL/RE expert will read the full text to understand how these were computed, most first time readers will skip through sections without pictures.

The links to ontology and benchmarks are provided, therefore the paper generally meets the criteria outlined by the editorial team.

Overall I like the paper, but would still rather suggest Minor Revision to allow the authors to correct the various minor things I noted here, as I do think that papers need to be addressed to both experts and casual readers.

## General comments
In this paper the authors introduce the I-ADOPT benchmark, an expert annotated corpus of 102 scientific variables represented in a the form of a machine-readable knowledge graph, grounded to the I-ADOPT ontology. In addition to this community resource, the authors conduct a detailed analysis on the use of LLMs to automate the generation of these semantic scientific variables.

The introduction clearly describes the challenges associated to the interpretation of measures associated to scientific variables reported in research tabular data with a concrete example. The I-Adopt ontology is then presented as a way to increase the semantic interoperability of scientific variables, however several major issues remains since i) scientists may have different but correct ways of representing variables, although they follow the ontology structure, and ii) it requires time and technical expertise. The related works section covers research activities aimed at representing scientific variables (SOSA/SSN, Schema.org, Croissant) as well as knowledge extraction from text approaches. The I-ADOPT model is then introduced as previous work and background. The main contributions consists in the description of the I-ADOPT corpus and the analysis of LLM-based RDF triples generation in the respective "Corpus" and "Benchmark" sections. Performance results are presented finally through an automated evaluation of the many studied experimental setups as well as a qualitative expert-based evaluation. Results show that the scientific variable representation through I-ADOPT is still challenging for the studied LLM setups, these results are interpreted in the Discussion section. Three future works are finally presented to overcome the identified limitations.

I liked reading this paper. We assist nowadays in a plethora of approaches leveraging LLMs to curate and populate data/knowledge bases with expected high quality content. However validating the approaches is difficult and expert validated ground truth are urgently needed to effectively assess the viability and to what extent LLMs can be used, as well as to better understand their best setup, size, and interaction strategies. This is precisely the aims of this paper which make it a very timely scientific contribution. The results are precisely reported leading to an in-depth discussion and interpretation.

Despite the quality of this work, the reading of the paper is sometimes difficult. Illustrative examples would be very useful to understand why the generation task is so challenging. Some parts of the I-ADOPT model such as symmetric and asymmetric systems are also difficult to understand for the reader without illustrative examples. Regarding the benchmark description, a graphical representation of the LLM-based generation workflow would ease the reading of the 2.5 pages.

On a more technical side, it's not easy to understand why SHACL rules are used to validate the corpus, but JSON-schema is used within the LLM-based generation workflow. JSON seems to be the cornerstone within the generation process. Has JSON-LD been envisaged as an intermediate representation format ?

Regarding the experimental results, it seems that Qwen32B is not "that big" for the challenging decomposition of scientific variable. Have the authors tested on larger models such as gpt-oss-120b, llama-3.3-70b for open source models, or even a state-of-the-art large proprietary model. A qualitative evaluation on a subset of the corpus (even with only 3 easy/medium/complex variables) would have been interesting to complement the study. In the conclusion, the scalability of the approach is presented as a potential issue. How costly is the whole annotation workflow ? Reporting estimations of the volume of generated tokens, and size of the context / prompts would have been useful to discuss the scalability of the approach. The results reported in Tables 4 and 6 do not show a noticeable improvement when introducing specific generation strategies in the prompt. This should be better discussed.

Finally on the data availability, [https://doi.org/10.5281/zenodo.15313672](https://doi.org/10.5281/zenodo.15313672) do not refer to the I-ADOP Corpus or Benchmark. It seems to be related to a research survey (LLM4OE_SLR).
## Detailed comments

### Related works
- Although it is stated that I-ADOPT is already adopted by a large community, the positioning with respect to Schema.org or SOSA/SSN (W3C) could be enhanced.
- Schema.org can rely on DefinedTerms to associate domain specific concepts, in additions the range of some properties are very permissive in terms of "expected types".
- What about context and provenance metadata associated to observations in I-ADOPT ? These kind of metadata could also be considered to increase the potential data reuse. SOSA/SSN seems to be partly overlapping with Fig 1. Is it possible to align I-ADOPT with SOSA/SSN ?
- "This survey reviews more than 4000 papers". The cited paper has 205 references and was partly generated by an LLM.
- "Text2Bench" → Text2KGBench
### I-ADOPT model (background)
- Examples are needed to grasp the underlying idea of "complex" "symmetric" / "asymmetric systems" as well as "fluxes". Figure 1-2 are missing the illustration of source/target.
- We don't see how QUDT can be used in Figure 2 to represent the units of the i-observations/measures.
### I-ADOPT corpus
- "Constraint annotations were omitted for the same reason" → not easy to follow
- Expert evaluation scheme: in addition to "understandable" why the experts are not ask if they would be able to actually reuse a variable ?
- A link to the actual corpus should be indicated in this section.
### I-ADOPT benchmark
- As said before, a diagram for the whole workflow would be useful here
- "to rely exclusively on the provided variable definition" → are the authors 100% sure that the LLM is not relying on it's "memory" ?
- For populating an ontology like I-ADOPT, it seems counter-intuitive to not rely on RDF serializations or SHACL. Would JSON-LD be useful ? Why not relying on SHACL for schema conformance ?
- An example would help in understanding why the "matrix_decision_tree" strategy is useful. Also, the use of "decision tree" is a bit misleading since it a set of ordered instruction, not especially structured as a tree.
- "distinguishing matrix-like contexts from constraints" → unclear
- "Systems are treated as representations" [...] "asymmetric .. symmetric ..." → this should be clarified or removed
- "Asymmetric systems" → missing examples, what is the difference between a ratio ? is it completely recursive ? what kind of mathematical formulae are supported ?
- The reranking part is clear and detailed.
### Results
- Table 3 is missing bold characters for the most salient results
- Why using "slot" instead of "property" for the I-ADOPT ontology ?
- Interpretation of GT and PRED conformance metrics is quite unclear. Is it a similarity distance between the textual definition of the variable ?
### Discussion
- A clear justification of how challenging is the task
- Reducing the subjectivity of variable definition is an interesting but not mentioned as future works
- An example of reusable pattern could described, based on the 102 GT variables
- "Combining these patterns with decision trees to guide the process, LLMs are expected to perform better", this has to be demonstrated since the provided "decision trees" in table 4 and 6 do not show noticeable improvements.
- "flows", "asymmetric systems", again, hard to follow since no examples have been provided
- "has observed in the ground truth", should refer to a clear example
- the semantic evaluation refers to the experts evaluation, and not the automated evaluation ? this should be clarified ("the two evaluations")
### Conclusions
- Key messages are clearly and precisely stated as well as future works
- "prompt engineering improves performance", again this statement is not in line with table 4 and 6
- "... remains intrinsically difficult for language models" and "Implicit knowledge may penalize .." these two messages seems to be overlapping

The paper introduces a benchmark for evaluating whether LLMs can decompose natural language definitions of scientific variables into structured I-ADOPT representations. It contributes an expert curated corpus of 102 variables across several domains, a decomposition task with a JSON schema driven prompting setup, an entity linking task against Wikidata, and a per-slot error analysis. The best model (Qwen3-32B) reaches F1exact ≈ 0.45 / F1close ≈ 0.46, and the authors conclude that LLMs alone are not yet sufficient and that formalized patterns plus human validation remain necessary. The problem is well motivated and the corpus is a genuine, reusable contribution. My main reservations concern several numerical inconsistencies in the corpus description, a JSON schema in the appendix that appears to be invalid, evaluation robustness under a non zero sampling temperature, and the absence of a non LLM baseline. None are fatal but thy need to be resolved before acceptance

Strengths
----

The corpus is the strongest part of the work : expert curated, multi domain, published with Turtle files and a browsable catalogue, and tied to GitHub discussions, a real resource for a community that currently lacks one. The bench is transparent (prompts, schemas, scripts, shot settings and per slot metrics are all reported) which I appreciate for reproducibility. The per component analysis is more informative than a single aggregate score and correctly localizes where models fail. The paper is also honest about its negative result rather than overselling the models.

Major limits
---

1. Several counts do not reconcile. Page 7 states 34 variables were accepted without changes and 77 required adjustments, which sums to 111, not 102. The constraint counts (85 with, 16 without) sum to 101, as do the matrix counts (61 with, 40 without). Individually minor but for a resource presented as a gold standard they undermine confidence in the curation. Authors should reconcile all corpus level counts against the correct total and state the denominator explicitly in each case

2. Table 2 is captioned "Percentage of 102 variables," but the values (46.32, 26.32, 9.47, 23.16) are not integers over 102 - 46.32% × 102 = 47.25 variables ---> which is impossible. They are consistent with a denominator of 95. Please state the actual denominator and explain how these categories relate to the "21 variables decomposed directly" figure mentioned in the same section

3. In Appendix ( p.18), the JSON schema for AsymmetricSystem appears invalid. in entityOrSystem, the asymmetric-system branch lists hasSource, hasTarget, hasNumerator, and hasDenominator as required, but properties defines only AsymmetricSystem, hasSource, and hasTarget. Combined with "additionalProperties": false, a conforming object is impossible to produce: hasNumeratorand hasDenominator are simultaneously required and forbidden. Since the ratio example in Figure 2 relies precisely on hasNumerator/hasDenominator, this could mean the model could never emit a schema-valid asymmetric system, which would directly bias the very low Matrix/system results and the expert assessment. Please confirm whether this is the exact schema used in the experiments. if so, the system related results need to be rerun and reinterpreted. if it is a transcription error, please correct it and confirm the experimental schema differed.

4. Evaluation robustness under temperature 0.5. The best configurations use temperature 0.5 but results appear to come from single runs, so the reported F1 values may reflect one particular decoding. authors shoud repeat at least the best Qwen332B and Qwen38B configurations several times and report mean +/- standard deviation and justify why a single run is adequate. Without this the "< 4% gain from 8B to 32B" claim rests on unquantified variance.

5. the discussion concludes that formalized patterns and decision trees are needed, yet the benchmark includes no rule based, pattern based or NER/RE baseline. A simple ontology guided or pattern matching baseline would both contextualize the LLM scores and support the paper's own central claim. Authors must add at least one such baseline.

6. the expert validation protocol is underspecified. 15 experts are mentioned but it is unclear how many variables each evaluated, whether variables were multiply annotated, and whether inter annotator agreement was measured. For Tab 1 to support a gold standard, authors should report the annotation design and an agreement statistic. Relatedly the final semantic assessment (Tab 8) appears to have been done by a single expert who cobuilt the corpus, a potential source of bias that should at least be acknowledged, ideally mitigated with a second annotator on a subset.

7. The conclusion that 8B --> 32B yields less than 4% improvement holds only for aggregated F1close (0.42 --> 0.46). For F1exact it is 0.38 --> 0.45 (+7 points), and per slot gains for ObjectOfInterest and Constraint are larger (Tab 5). Please qualify the claim to specify the metric since as written it understates the effect of model size.

8. the corpus is dominated by Earth and Environmental Sciences (40/102), yet the paper claims a domain independent benchmark. Without per domain results it is hard to tell whether the scores measure general decomposition ability or proximity to the founding community's variables. Please report results broken down by domain anddiscuss this limitation explicitly.

9. The motivating example (Fig 2) uses QUDT, NERc and PATO but the linking evaluation (Tab 7) is Wikidata only which narrows its practical relevance. Either extend the evaluation to the vocabularies actually used in the corpus or state this scope limitation clearly.

Other minor limits :
---

The hasMatrix sentence on p 13 is ambiguous and likely incorrect: "The system representation for entities was used only once in the ObjectOfInterest slot (29 occurrences in the corpus) and twice in the Matrix slot (7 occurrences in the corpus)." This contradicts p.7 (27 systems in the matrix slot) and the parenthetical counts are unclear. Please rewrite

Authors must state explicitly how aggregated F1 in tab 3 is computed, since the global F1 is not simply derivable from the reported precision and recall. Spelling out the per slot or per variable averaging would prevent readers from suspecting an eror.

Tab 8 labels ("Good / Acceptable / Weak / wrong") don't match the grade definitions in the text ("Good / Correct / Weak / Wrong"). Align the labels and capitalize "wrong."

Typos:
--

"strenghts" --> "strengths" (p 2)

"representants of SVO" --> "representatives of SVO" (p 2)

"biochemstry" --> "biochemistry" (p.4/5)

"Resarch Infrastructure" --> "Research Infrastructure" (p.3)

"the The I-ADOPT Corpus" --> duplicate "the" (p 2)

"Errors Analysis" --> "Error Analysis" (p 11)

"na¨ıve" --> should render as "naïve" (p 11, Table 7)

"AGUFMA41L" footnote 30 and a few references have broken accents/encoding (e.g. "Lefranc¸ois") worth a pass.