Testing Prompt Engineering Methods for Knowledge Extraction from Text

Tracking #: 3719-4933

Authors: 
Fina Polat
Ilaria Tiddi
Paul Groth

Responsible editor: 
Guest Editors KG Gen from Text 2023

Submission type: 
Full Paper
Abstract: 
The capabilities of Large Language Models (LLMs,) such as Mistral 7B, Llama 3, GPT-4, present a significant opportunity for knowledge extraction (KE) from text. However, LLMs' context-sensitivity can hinder obtaining precise and task-aligned outcomes, thereby requiring prompt engineering. This study explores the efficacy of five prompt methods with different task demonstration strategies across 17 different prompt templates, utilizing a relation extraction dataset (RED-FM) with the aforementioned LLMs. To facilitate evaluation, we introduce a novel framework grounded in Wikidata's ontology. The findings demonstrate that LLMs are capable of extracting a diverse array of facts from text. Notably, incorporating a simple instruction accompanied by a task demonstration—comprising three examples selected via a retrieval mechanism—significantly enhances performance across Mistral 7B, Llama 3, and GPT-4. The effectiveness of reasoning-oriented prompting methods such as Chain-of-Thought, Reasoning and Acting, while improved with task demonstrations, does not surpass alternative methods. This suggests that framing extraction as a reasoning task may not be necessary for KE. Notably, task demonstrations leveraging examples selected via retrieval mechanisms facilitate effective knowledge extraction across all tested prompting strategies and LLMs.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 08/Jul/2024
Suggestion:
Accept
Review Comment:

The work makes an important contribution to the intersection of Large Language Models (LLMs) and testing rapid engineering approaches for knowledge extraction from text utilizing LLMs such as GPT-4. The authors addressed the comments and revised the text accordingly. Furthermore, the results were extended to include an evaluation of RAG in the analysis.

Review #2
Anonymous submitted on 28/Jul/2024
Suggestion:
Accept
Review Comment:

This is the second review iteration of this manuscript and the authors addressed all the remarks from my initial review. The additional work that the authors added in this version, that is:
- evaluate the methodology with 2 extra open LLMs,
- details on the post-processing process and,
- discussion on the analysis of the extraction
further strengthens this work.