Towards Explainable Knowledge Graph Embeddings by Respecting Logical Commitments

Tracking #: 3051-4265

Authors: 
Mena Leemhuis
Oezguer Oezcep
Diedrich Wolter

Responsible editor: 
Guest Editors Ontologies in XAI

Submission type: 
Full Paper
Abstract: 
Knowledge graph embeddings (KGEs) can be seen as opportunity to integrate machine learning (ML) with knowledge representation and reasoning. In KGEs, concepts and relations are represented by geometric structures that are induced by ML. Explicit representation of concepts and relations empowers reasoning which can augment ML. This additional symbolic layer linked to ML models is widely advocated to foster explainability. However, symbolic reasoning and ML need to be aligned beyond the level of concept symbols in order to obtain explanations of what was actually learned. We characterize explainability as the alignment of reasoning in two agents, which calls for a rigorous understanding of reasoning grounded in KGEs. The desired alignment can be achieved by investigating the logical commitments made in KGE approaches by identifying models of logics that are aligned with ML models. Not until logical commitments of KGEs are aligned with common modes of reasoning, explanations for learnt models can be generated that are both effective and semantically congruent with what has been learnt. We critically review existing approaches to KGEs and then analyze a cone-based model capable of grasping full negation, a property common to symbolic reasoning but not yet captured in current KGE approaches. To this end, we propose orthologics as basis to characterize cone-based models.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 04/May/2022
Suggestion:
Reject
Review Comment:

The paper considers explainability as model alignment two agents. Roughly speaking, an agent 1 can explain itself to another agent 2, if one can equip them with a logic L1 and L2 respectively, and the queries in L2 and their answers in L1 can be translated back and forth.

This paper is an investigation of the choices to be made when equipping the agents with a logic.

I have been pondering about this submission for a long time.

I came to the conclusion, and it is an opinion, that it is a paper with great ambition but with imperfect execution. I think that these 15 pages are touching upon too many complicated topics all at once. KG embeddings, explainability as model alignment, a newly introduced notion of logical commitment, geometric structures, orthologics and subortholattices, ...

The notion of logical commitment presented in Section 2 is not altogether clear. It is also in this section that "explainability" is defined. This definition is rather clear to me. But the exact connection to logical commitment is not.

When Theorem 1, Proposition 1, and Table 2 arrive towards the end of the paper I would expect that everything would be tied in. Instead I am left with zero understanding of what they mean. I am not completely illiterate in things of formal logic, but I can honestly cannot say whether Section 4 is sound or gibberish, and I am not capable either of linking it to the problem of explainability.

It may be too ambitious for a 15-page paper.
It may be just too complicated for me. I mean, it is too complicated for me. Maybe it is just that.

If I may suggest something to the authors, it would be the presentation dumb it down. (Not the content of course.) Unless the paper is targeted at specialists of the abstract theory of KG embeddings, I am convinced that if accepted as is, this paper will not meet its audience.

- p 4, l 34, equation: KB should be A1?
- p 6, l 6: "resolving the ubiquitous expressivity vs. feasibility dilemma strictly in favour of feasibility" That's not resolving the dilemma.
- p 9, l 48: "are allowed to be arbitrary pairs of vectors" sets of pairs of vectors?
- p 11, l 2: mark better the separation between 4th and 5th axiom: A -||- ~~A A & ~A |- B
- p 11, p 35: \orthoneg is over X? I think A^\orthoneg should be {x \in X | x \orthneg y \forall y \in A}

- p1, abstract: "can be seen as opportunity" -> as an
- p 2, l 12-21 is hard to parse
- p 2, l 32: "can be seen as opportunity" -> as an
- p 2, l 44: "logical commitment as an advancememt of ontological commitment" what do you mean by "advancement"?
- p 3, l 10-11: "both terms a interrelated" are
- p 4, l 36: comma after "in the above sense"
- p 4, l 36-43 unclear.
- p 4, l 46: "as wells" no s
- p 5, l 27-29 is hard to parse
- p 7, section header 3.1: "Logics Grounded in Geometric Structues" structures
- p 8, l 18 "an query (complex) Boolean query q" a (complex) Boolean query q?
- p 8, l 51: "allow to" grammar
- p 10, l 49: "orthocomplent" orthocomplement?
- p 11, l 23: "(cp. Table 1)" cf?
- p 11, l 36: "where where"
- p 11, l 39 "see also Figure 3, bottom" where is the bottom?
- p 14, l 26 "cp. Definition 1" cf?

Review #2
By Till Mossakowski submitted on 15/May/2022
Suggestion:
Reject
Review Comment:

The paper consists of two halves: in the first half, logical commitments
for knowledge graph embeddings are examined. There is a certain interaction
between the geometry of the embedding and the logic underlying the embedding.
A related survey is [7]; however, it reviews only approaches based on
threshold functions, which geometrically means that relations are
expressed via threshold balls in Euclidean space. By constrast, the
present paper takes a much broder view.
This examination culminates in Table 1, where eight different
approaches are listed with their geometries and logical
commitments. This is clearly an interesting survey, based on different
levels of logical commitments (Fig. 1), like different operators or
calculi.
However, the exact relation between geometry and logic remains quite
vague here. Specific examples are discussed only briefly. The different
underlying algebraic structures are not discussed in detail (with the
exception of ortholattices), and in Table 1, only some rows list
the corresponding algebraic structures.
Morevoer, the authors speak about logics for TBox, ABox, querying and
explainability (involving three different logics L, L_1, L_2), and
have some general considerations, but it is unclear how these general
considerations influence Table 1. For exmaple, in Table 1, only logic
L is specified - what about L_1 and L_2? Also, the general
considerations themselves remain too vague in various respects. For
example, the authors consider the knowledge based of all L_1-sentences
true in I_E, where L_1 is the main logic in use. What would this
amount to in the examples? Is this knowledge base really relevant for
knowledge graph embeddings?
The authors introduce a definition for the central notion of the special
issue, explainability, but the definition does not make much sense in my
eyes.
Basically, at various places, the introduced notions and notations need
more explanation and examples in order to be digestablle. Moreover,
I doubt that the authors themselves have thought things through.
See also the detailed comments.

The second half of the paper examines orthologics, i.e. logics for
ortholattices and their relation to geometric models based on
cones. It is unclear to me why this specific embedding approach (which
clearly deserves its place in Table 1) deserves so much attention
here. The authors write "This will give us an opportunity to motivate
orthologics as framework to compare different approaches." (p.10
l. 26f) However, I cannot see that this claim has been
fulfilled. Moreover, even if one ignores this lack of motivation, one
must say that the paper does not develop this approach very well
w.r.t. its overall context. No knowledge graph embedding based on
cones and orthlogic is presented, let alone evaluated. Instead, the
authors conduct a purely theoretical investigation of orthologics and
cone-based ortholattices, culminating in Table 2, with no link to
knowledge graph embeddings, nor explainability.

Hence, I think that this paper canot be accepted. I suggested that the
authors split the paper into two papers. These both need substantial
revision before they can be resubmitted.

Detailed comments:

p.2 l.1 must play -> should play (otherwise, "at least" sounds a bit
peculiar here)

various places: logic properties -> logical properties

p.2 l.14 a KGE

p.3 l.27ff.
It reamains unclear why ortholattices receive so much attention here.

p.4, Def. 1
You defined what it means that agent A_1 is explainable to agent A_2,
but in the definiens, A_1 does not occur, except from logic L_1
being supported by A_1. This does not make sense. Probably, A_1
needs to be related to the knowledge base KB somehow?

Moreover, the assumption that there is a common super-logic L of
logic L_1 (used by agent A_1) and logic L_2 (used by agent A_2) is
very strong.

Finally, the most intersting case of explainability is not covered
by Def. 1: namely the case when A_1 uses a subsymbolic system like
a neural network, and A_2 explains A_1's answers using some logic.

p.4 l. 45
"The above definition presumes a notion of logic that comes with a
syntax and semantics with the usual notions of
structures/interpretations and of models as wells as derived notions
such as entailment which can be used to formally define query
answering."
No, Def. 1 only presumes a notion of query answering. This could be
purely proof-theoretic or alorithmic, with no semantics. (Of course,
a semantics is highly desirable, but not presumed by Def. 1.)

p.4 l.48
"which gives rise to some logic L'_1" why? how? example?

p.5 l.27
logical properties

p.5 l.50, p.7 l. 31
use "how ... look" or "what ... look like", but not "how ... look like"

p.7 l.12
this is not really a union, more a tupling

p.7 l.22 The king-queen-type of word puzzles is more a type of analogy
reasoning, and not the usual kind of querying based on logics. Vector
arithmetic in word embeddings supports these types of analogies. It is
an open question how to capture this logically. But
1. this type of open question does not make sense at this place
2. the authors' reference to [30], a paper on mathematical induction (and not
induction in the sense of learning), does not make any sense here.

p.8 l.5 "sets concept symbols" - what is this?

p.8 l.25 "I ∈ gMod(EA) is true w.r.t. the pair (T , A)"
This does not make any sense to me. What is the relation between EA and
(T,A)? Probably you need something like gMod(EA,(T,A)) ?
This definitely needs more explanation and examples.

p.8. l.29 consistency in FOL is not semidecidable, only co-semidecidable
(that is, inconsistency is semidecidable).

p.8 l.32 "query answering amounts to model checking"
only for ground queries. The interesting case, queries with variables,
amounts to finding an answer substitution, which is more complex
than model checking.

p.8 l. 39f. The authors speak of embedding structures E and
"associated logical structures I_E" without defining what this
means. Probably, the construction of I_E depends very much on the
embedding? Then at least one or two more detailed examples would be
needed here.

p.8 l. 44f.
"Of course everything hinges upon finding
appropriate logics L, L_1 , L_2 and such that L_1 is sound and
complete for the class of structures of an embedding.",
It is not clear what soundness and completeness would mean here.

p.9, Table 1
The "Operators" column could be more precise.
E.g. "sub-Boolean algebra"s - are these e.g. inf-semilattices?
Is negation as failure algebraic at all?
The row with "threshold balls" is a bit imprecise. Note that there
are approaches like TransR [1] that use a projection before applying
a threshold ball. Would this be useful for other approaches, too?
[1] Yankai Lin, Zhiyuan Liu, Maosong Sun1, Yang Liu, Xuan Zhu:
Learning Entity and Relation Embeddings for Knowledge Graph Completion

p.9 l.45 superfluous ")"

p.10 l.39 How can orthologics be classic, if they do not obey the
law of excluded middle? E.g. Kleene's three-valued logic should be a model.

p.11 l.2 I do not udnerstand the fourth axiom. How can A be equivalent
to ~~A&~A ???

p.11 l.22 non-orthomodular: this has not been defined

p.11 l.35 lowercase x in the formula

p.11 l.37 What is the "resulting logic"?

p.11 l.38 orthomodel: this has not been defined

p.12 Prop. 1
Where do you prove that this is an orthoframe?
In the proof, you tacitly use a characterisation of the closure of A
that should be made explicit.

p.13 l.46 "hence must be a Boolean algebra" - how does excluded middle follow?

p.14 l.4-8: Here you try some link to explainability. However, the link
remains completely unclear to me.
Moreover, no relation of orthologics to knowledge graph embeddings is
provided nor discussed.

Review #3
Anonymous submitted on 06/Jul/2022
Suggestion:
Reject
Review Comment:

The paper discusses a method for explainability of knowledge graphs embedding approaches on the base of the notion of "logical commitment". After a revision of the current KGE approaches in a common model, the authors present the characterization of such unifying model in orthologics.

The topic of explainability in the context of (symbolic and subsymbolic) reasoning methods for Knowledge Graphs are clearly of interest for the journal and to the XAI Special Issue.

On the other hand, the work in its current state has several problems regarding the clarity of the formal methods, the contents of contributions and evaluation of the method.

The main issue with the paper stands in the substance of the contributions and the evaluation of the proposed formalization: as admitted by the authors in the conclusions, the paper appears only to be a "first roadmap" of the aspects that need to be investigated more than a full-fledged study, which makes it not sufficient for a journal publication.
The need for logical commitments and the realization in orthologic do not have sufficient justification in the presented submission.
The lack of a clear assessment of the proposal makes it also difficult to assess the possible impact and applicability of the proposed characterization.

With respect to the writing and organization of the paper, the formalization should be presented with a more clear distinction of definitions in the text and the necessary preliminaries (see for example the use of "agents" notation in Def.1, which have not been introduced previously, or the definitions for KG triples).
The few formal propositions (on p.12) are discussed and explained very briefly.

The same notion of logical commitment (as introduced in Section 2) is not entirely clear: it appears to be related to the choice of the language of interest, but the axioms defining the semantics are then considered "nuances". In considering the calculus of interest, no mention is given with respect to the complexity of reasoning in a specific logic. It is not clear why algebraic ortholattices are then used as the formalism of choice.

The details of the presentation of the formal aspects is not enough for replicating the method e.g. applying it to other KGE approaches, as it is given at a too high level.