Ontology supported semantic based image retrieval

Tracking #: 3660-4874

Akif Gasi
Tolga Ensari
Mustafa Dağtekin

Responsible editor: 
Cogan Shimizu

Submission type: 
Full Paper
In this study, a two-stage approach for developing a Semantic-Based Image Retrieval system supported by Ontology is proposed. In the initial stage, the Object Detection process is employed to identify objects within the image. Subsequently, a predicate describing the relationship between these two objects is determined using the developed Bi-directional Recurrent Neural Network (Bi-RNN) model. In the performance measurement of the developed model, 91% accuracy was obtained according to the Recall@100 (Top-5 accuracy) result. In the second stage, relations defined in the form of <subject-predicate-object> are transformed into Ontologies and utilized to search for images that are semantically similar. In addressing the primary challenge of Semantic Gap within the Semantic-Based Image Retrieval approach, the proposed solution involves measuring the number of similar relationships between two images through the utilization of entropy. The Semantic Gap between two images was computed using the Joint Entropy method, leveraging the number of relationships (X) identified in the query image and the total number of relationships (Y) in the image with similar relationships obtained as a query result. The proposed approach exhibits characteristics of a novel method within this field, distinct from other similar methods employed in Semantic-Based Image Retrieval through the utilization of Ontologies.
Full PDF Version: 


Solicited Reviews:
Click to Expand/Collapse
Review #1
By Jean Vincent Fonou-Dombeu submitted on 26/Mar/2024
Minor Revision
Review Comment:

1. the sentence added in the abstract to report the stats should be written a the end of the abstract not a the beginning.
2. The authors must explain the variables in Equations 1 t0 4 below the equations to enable the reader to understand these equations
3. The paper must be proof read by a native English speaking person.
4. Table 1 does not have a title; the authors have explained the Table in lieu of the title; the explanation must be moved and a title provided.

Review #2
By Md Kamruzzaman Sarker submitted on 29/Mar/2024
Review Comment:

The authors have resolved most of my comments and the paper has been improved. It can be accepted.

Review #3
Anonymous submitted on 14/Apr/2024
Review Comment:

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data, (B) whether the provided resources appear to be complete for replication of experiments, and if not, why, (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.


The paper proposes a novel two-stage Semantic-Based Image Retrieval (SBIR) system supported by Ontology, aiming to address the Semantic Gap in traditional Image Retrieval approaches. In the first stage, Object Detection and a Bi-directional Recurrent Neural Network (Bi-RNN) model are employed to determine relationships between objects. The second stage involves converting these relationships into Ontologies for more effective semantic similarity searches. The study introduces a method for measuring Semantic Gap using entropy.


Novel Ontology-Supported Approach: The paper introduces a unique two-stage SBIR approach that utilizes Ontologies for improved semantic representation and retrieval.

Effective Use of Visual Genome Dataset: The study employs the Visual Genome dataset for training the model and generating ontologies, contributing to the credibility of the research.

Innovative Semantic Gap Measurement: The use of entropy, specifically the Joint Entropy method, for measuring the Semantic Gap between images adds a novel quantitative dimension to the evaluation.

The paper demonstrates a high level of originality by combining Object Detection, Bi-RNN models, and Ontology in a novel two-stage SBIR approach. The introduction of entropy for Semantic Gap measurement further contributes to the originality of the study.

Significance of Results:
The proposed approach, especially the second stage utilizing Ontologies, is shown to yield more effective results in semantic similarity searches compared to existing methods. The introduction of a quantitative measure for Semantic Gap provides a valuable contribution to the field.

Quality of Writing:
The writing is generally clear and concise, with well-organized sections. The technical details, such as the Word Embedding process, Bi-RNN model description, and Ontology creation, are presented in a detailed manner.


Weak points have been addressed in the revised version, manuscript can be considered for acceptance.