Evaluating Knowledge Graph Construction from Text Without Supervision

Knowledge graphs (KGs) are essential for organizing and retrieving knowledge from large text collections, yet constructing high-quality KGs from text remains a difficult task. An important challenge when constructing KGs is evaluation: How can we measure the quality of the graph under construction? Traditional evaluation relies on expensive, manually curated gold-standard graphs whose creation demands significant expert effort and is rarely available for every domain. To alleviate this cost, we need unsupervised metrics that compare generated graphs directly with the source texts. Unfortunately, most proposals in this direction require training or fine-tuning machine learning models, introducing additional costs that limit their applicability. To address this gap, we develop a fully unsupervised metric that compares graphs directly with the source texts. Our proposal, called Bidirectional Graph Similarity (BIGS), transforms each triple into natural language sentences and measures semantic alignment with original document sentences using embedding similarity. We evaluate BIGS in several document collections and standard benchmarks for supervised evaluation, finding a strong correlation with supervised metrics, and that it remains robust across different text segmentation and verbalization strategies. Our results show that BIGS effectively captures the coverage of generated graphs without any manual annotations or additional training, offering a practical tool for KG evaluation.