A Systematic Survey of Semantic Web Technologies for Bias in Artificial Intelligence Solutions

Overall evaluation

The topic of the paper is ‘hot’ and its content is definitely worth publishing it. However, due to presentation issues (mainly) it must be significantly revised. The structure of the paper could be improved, the language (syntax/grammar) is also a weak point that must be fixed. Regarding the survey methodology followed, there are a few points that need attention (pls see detailed comments). Finally, examples are missing from the paper; a few insightful examples on AI bias and bias in KGs would certainly improve readability and comprehension.
According to SWJ review criteria for survey articles, the following is my view and understanding:
(1) The paper is suitable as an introductory text, targeted at researchers, PhD students, or practitioners, to get started on the covered topic. However, insightful examples are missing, and language problems must be fixed.
(2) The presentation and coverage of the topic is comprehensive and balanced, but it can be significantly improved.
(3) In terms of presentation, the readability and clarity of the paper can be significantly improved.
(4) There is no doubt that the paper is of high importance to the broader Semantic Web community.
I would like to thank authors for contributing this survey to the SW/AI research community.

Detailed Comments

Title: The use of 'for' in the title is somehow vague or misleading. IMHO I believe 'and' would better reflect the interrelation of SW and AI bias.

Abstract: Revision is needed according to detailed comments.

Page 1:
- Pls avoid citing sources in your abstract.
-“to bringing solution…” -> 'to solve the problem of ...' but which specifically problem? the bias or data validity?
- ‘fulfil’ -> 'bridge' might be a better term here

- ‘…, there exists no previous work to bring together bias and semantics’ -> so, what about 'bias in KGs'? is this not related? There are plenty of existing works that are related to this topic. Also, I do recall a paper (Debiasing Knowledge Graphs: Why Female Presidents are not like Female Popes) by Janowicz et al. 2018 about this topic, especially linking bias at schema, data and reasoning level. Page 16 (Bias within semantic resources) is also referring to related works that do 'bring together bias and semantics'. So, I guess this statement should be corrected somehow.

‘…types and sources of bias addressed with…’ -> bias itself? you mean probably 'bias assessment or/and 'bias mitigation' or 'bias interpretation' (to be accurate) or something else e.g., bias representation?

-‘… improve frequent limitations in AI systems.’ -> such as? pls give an example here

- ‘…We find works…’ -> we 'research works' maybe?

Keywords

-conceptual semantics’ -> I would prefer the term 'conceptual model' or 'semantic model' or just 'semantics', since 'conceptual semantics' has been used in the past (mainly) by Ray Jackendoff as a framework for semantic analysis. Anyway, you also use this term (semantics) in several places in the paper.

1. Introduction: Please revise according to the detailed comments. Also, please provide examples of biases where suitable, for readers to be able to get a first idea of what you are presenting in this paper (what do you try to tackle). Examples are missing! (Note: A few good examples of KG bias at different levels are presented in Janovicz et al. 2018 (Debiasing KGs...).

Page1:
Line 34: ’However, other factors coming from the humans’ -> which ones? Pls be specific.
Line 39: ‘to fill in existing gaps of AI systems,’ -> which are? Pls be precise.

Page2:
Line 5: ’conceptualisation that accounts for these dimensions.’ -> which dimensions?
Line 15: ‘for developing solutions to different biases’ -> 'solutions to bias' must be explained further i.e., it is 'bias assessment solutions' or 'bias mitigation solutions' or 'bias interpretation solutions' or other
Line 25: ‘literature of semantics and bias.’ -> pls use the term ‘AI bias’ for precision
Line 33 and 34: ‘Section 6 is’, ‘Section 7 are’ -> pls rephrase and correct syntax/grammar

2. Background of conceptual semantics: Pls revise according to the specific details. Overall, this section should be enriched/extended with more background knowledge, beyond semantics. One such extension is the 'AI bias' topic. What is AI bias? How is it assessed? How is it mitigated? What is bias interpretation/explanation? What is fair AI?

Page 2:
Line 4: ‘SKOS’ -> As you know, SKOS is the technology to represent taxonomies, but an example of a taxonomy would be useful here also e.g., Yahoo taxonomy, AGROVOC taxonomy,…
Line 13: ‘Ontologies.’ -> Example ontologies are missing. Also, an additional sentence or two about the different types of ontologies would be insightful for readers.
Line 15-17: ‘conceptualisation that is characterised by high semantic expressiveness required for increased complexity.’ -> but what about lightweight ontologies? These are ontologies as well... I suggest to just keep Gruber's definition (or use one of other more recent ones).
Line 22: ‘A knowledge graph (KG) is based on’ -> why 'based on' and not 'is...'? There are many definitions of KGs today to cite here.
Line 32: ‘Linked Data.’ -> This is a weak description, mainly because of what I have provided as argument in the use of KB, KG and LD terms earlier in this section. If you keep it, I do not see how you can avoid mentioning RDF paradigm. Also, examples must be provided, as done in the other types of 'conceptual semantics'. By the way, it seems that a paragraph on KB is missing (I guess because you do understand its similarity to a KG or because of low importance in this context.
Line 36: ‘Many data objects’ -> I do not understand the use of this term and how it can 'fit' in the provided definitions you mention (which definitions, by the way?). I believe this paragraph needs elaboration or/and disambiguation.
Line 44,45: ‘ knowledge bases, knowledge graphs, and the linked data.’ -> I understand how and why you provide this distinction, however pls notice that a KG may be considered a KB, and a KG may be considered (a representation of) linked data. In my view, I would just use the term KG. Anyhow, KG is the key SW technology in respect to AI bias (as you also conclude in this paper).
Line 46-48: ‘There exists…[20]’ -> pls rephrase, not correct syntax

3. Survey Methodology: Pls revise according to the detailed comments. Also, pls consider extending the reviewed papers list with papers published in 2021 (first semester) due to the highly dynamic aspect of the topic.

Page 3:
Line 4: ‘this study is bias’ -> pls refer to 'AI bias' not just bias (all sections in the paper) to distinguish from other types of biases
Line 5,6: -> 'investigate' is used twice in the sentence, pls rephrase
Line 6: ‘the utility’ -> pls replace with ‘use’
Line 8: ‘which type of biases ‘ -> 'types of bias'
Line 10: ‘proposals and literature reviews)’ -> but why? Isn't this what you also have prepared to discuss the topic? If your review paper (when published) is excluded from future research, then all this significant knowledge you have provided will be ignored.
Line 11,12: ‘Finally, we explore how bias manifest to identify key challenges in AI’-> pls disambiguate/rephrase
Line 15: ‘Search string and’ -> ‘Search keywords’
Line 21: ‘Google Scholar’ -> What about Semantic Scholar?
Line 34: ‘3.3. Synthesis of the results’ -> Either elaborate/explain more this subsection or remove it (as it is now it does not contribute any important information in this section).
Line 43: ‘between 2010 and 2020.’ -> Since the topic (AI Bias and KGs) has been paid a lot of attention very recently, there are a few very promising related works already published in 2021 (most in arXiv.org as preprints). Pls have a look there also and append your list of related works.
Line 45: ‘as part of conference proceedings or workshop,’ -> Could you name a representative list? For instance, I trust that at least AAAI, IJCAI, ECAI, KR, ISWC and ESWC events (last 5 years) are included.

4. Dimensions of analysis: Pls revise according to the detailed comments. I would replace the title into 'Analysis approach'.

Page 3:
Line 49: ‘Section 4.2 defines different categories of bias according
to its type’ -> this is somehow redundant (categories=types), pls rephrase

Page 4:
Line 1: ‘ 4.2 Bias in AI categorisation’ -> Perhaps this subsection can be moved to Section 2 (Background knowledge), along with the knowledge on semantics.
Line 4: ‘definition as [4] of bias’ -> pls check syntax
Line 6-10: why in italics?
Line 19: ‘the bias types,’ -> ‘types of AI bias’
Line 21: ‘define their nature’ -> bias->its nature; biases-> their nature
Line 26: ‘4.1. Dimensions of conceptual semantic tasks’ -> The tile of this subsection is somehow misleading. Perhaps a better title is "AI bias and semantics".
Line 28: ‘main group of works’ -> 'groups of work'
Line 31: ‘Identifying bias’ -> Is this equal to 'assessing bias'? In any case (yes or no) pls state their relation. Also, replace ‘discover’ with ‘identify’ for consistency.
Line 36: At the end of paragraph, Pls provide an insightful example. In general, examples are missing from the paper. Please consider adding at least a few insightful ones related to the different types of bias.
Line 50: ‘based on a holdout set’ -> ??? pls explain

Page 5:
Line 1 to line 11 (left column): these two paragraphs seem not relevant to this subsection (2.2.2), pls check.
Line 10 and 18: Table 2 and 3 captions, please correct syntax (papers do not have bias type or origin…)
Line 12: ‘4.2.3. Bias impact’ -> the title refers to bias impact, but the content of this section presents bias types e.g., population bias, behavioral bias, etc.
Also, the use of word 'as' in several places in this subsection seem problematic in terms of syntax/grammar (and meaning eventually). Could you please check and disambiguate/fix?
Line 16: ‘challenges in some of the papers.’ -> which ones? Pls be specific (citations)
Line 30: ‘Two other categories’ -> which ones?

5. Analysis of results: Pls revise according to detailed comments. Not every syntax/grammar (language) issue is highlighted beyond this point (please ensure you check this and following section for related issues).

Page 5:
Line 42: ‘RS are aimed to discover’ -> syntax, pls rephrase (RS aim to recommend...)
Line 49: ‘specific methodology examples’ -> 'methodological'
Line 50: ‘could help extrapolate them’ -> pls use a synonym

Page 6:
Line 17: ‘ML groups works’ -> pls correct syntax and add references.
Line 19: ‘5.1.1.1. Bias at source (functional bias)’ -> four levels of subsections should be avoided for clarity and presentation reasons. Please consider re-structuring section 5. Perhaps a taxonomy or even a KG that represents the structure of the concepts presented in section 5 would be helpful.
Line 28: ‘NLP is used to comprise’ -> syntax
Line 34: ‘Finally, we refer as intelligence’ -> syntax
Line 45: ‘activity to group two groups’ -> syntax
Line 36: ‘and display of information’ -> ‘… and presentation of information’
Line 39: ‘Then, we introduce different AI system problems and’ -> syntax
Line 44-48: 5.1.1. -> this definition has already been provided (earlier section). Same for other subsections (5.1.2, 5.1.3)

Page 7:
Line 24: ‘user’s preferences to only the items mentioned’ -> syntax
Line 34: ‘word detectors to only the captions’ -> syntax

Page 8:
Line 17-19: syntax

Page 10:
Line 1: ‘5.2. Bias impact and use of semantics’ -> The most important section so far. In general, it is well written, however, a few points must be further elaborated, and arguments must be justified. Pls revise according to the detailed comments.

Line 7: ‘AI systems (Table 4). We only include in Table 4’ -> why you chose this approach? pls justify and argue on this. It seems (from the citations) that a number of related works are not researched.

Line 11: ‘We select the most appropriate papers’ -> what does 'appropriate' mean here? what is the criterion (or criteria) for this selection?
Line 19: ‘and therefore can be more representative’ -> not sure?
Line 28: ‘in [36] is based a’ -> ‘based on’
Line 48”: ‘We find’ -> pls rephrase e.g., A number of semantic approaches capturing bias… have been researched (pls correct this in all occurrences in the paper)
Page 11:
Line 1: ‘They proof’ -> ‘They prove’

6. Discussion: This section is the strong point of this paper. However, pls check for language problems (there are quite few) and fix appropriately.
Page 12:
Line 45: ‘at different stages of the AI pipeline,’ -> perhaps 'AI system/app pipeline'
Line 50: ‘Section 5 to bring to a discussion’ -> syntax

Page 14:
Line 6: ‘We see this in [4] and many other works,’ -> pls be specific (citations)
Line 41: ‘6.2’ title (and 6.3) -> pls change to a non-questioning title
Line 44: ‘To discuss with more’ -> discuss in
Page 16:
Line 31: ‘Bias within semantic resources.’ -> This topic is given little attention (space) in the paper; however, it seems to be (perhaps) the silver bullet in addressing AI bias using semantics i.e., KGs. If the used (external or internal) KGs are already biased, the rest of the process (identifying, mitigating, assessing bias) in AI models will be affected (include the already encoded bias of KGs), a kind-of bias-propagation effect.
Line 38: ‘bias can seek into’ -> ‘bias can be found’ perhaps? I might be wrong here, but in any case, the sentence needs disambiguation
Page 17:
Line 1: ‘However, this survey succeeds…’ -> pls remove ‘However’ and replace ‘succeeds’ with ‘aim to’
Line 4: ‘to help’ -> ‘to assist’
Line 10: ‘sense’ -> ‘meaning’

7. Conclusion
Page 17:
Line 17: ‘conceptual semantics to alleviate bias in AI.’ -> 'semantics' (eventually, since it has been used like this in the paper most of the times). Also, replace ‘alleviate’ with 'address'.
Line 30-32: ‘it. Comparing semantic methodologies to other state-of-the-art bias
mitigation approaches,’ -> where is this comparison presented in the paper? I might be wrong, but pls check and address appropriately.
Line 37: ‘use in the recent years of semantics,’ -> 'use of semantics in the recent years'
Line 38: ‘semantics are helpful to address’ -> 'are helpful in addressing' or 'help to address'

This manuscript was submitted as 'Survey Article' and should be reviewed along the following dimensions: (1) Suitability as introductory text, targeted at researchers, PhD students, or practitioners, to get started on the covered topic. (2) How comprehensive and how balanced is the presentation and coverage. (3) Readability and clarity of the presentation. (4) Importance of the covered material to the broader Semantic Web community. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data, (B) whether the provided resources appear to be complete for replication of experiments, and if not, why, (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.

Bias are too much nested with-in itself.
page-4. Section 4.2.1
Initially 3 types of biases: statistical, cognitive, cultural (from Yates)
Q1)In what way, these 3 bias types are related to semantics. Why this 3 types alone.

Yates mentioned that, cognitive and cultural bias are user-dependent.
Q2)How this user-dependent bias are linked to bias in semantic web(SW)?
No explanation given why this 3 biases are considered as core bias types in SW.

page-4, Section 4.2.2
"To understand the source of each bias, we follow Oltenau et al.’s [5] comprehensive framework....."
Bias at source, Bias at collection, Data preprocess, Data analysis
Oltenau et al.,[5] defined the biases and its limitations and pitfalls with respect to social data.
Q3)Why the biases defined for social data [5] is considered as source of bias for the SW?

---------------------------------------------------------------
page-5, Table-3
Classification according to bias origin: external, functional, sampling, querying, annotation, aggregation, inference
As per [5], only external and functional are biases at data source. Sampling, querying, annotation, aggregation, inferences are only activities/events that happens across the data pipeline which causes bias.
---------------------------------------------------------------
page-5, Table-3
Consider the references:
Bias at source - External bias - [30,36,49]
===========================================
Reference [30]: In tabe 5, page-13
classified under: Cultural -> External -> DL
It is mentioned as DL under AI system. [30] developed and released multilingual sentiment Ontology for sentiment and emotion detection in images.
Q4) Why it is not classified as NLP under AI system, since the dataset creation involved typical NLP processes defined in paper.

Semantics-?, Semantic resource-? ->Ontology not mentioned in table 5
---------------------------------------------------------------

Reference 36: In tabe 5, page-13
classified under: Cultural -> External -> MI -> Thesauri

AI -?
Q5) Why it is not classified as NLP under AI system, since [36] used WikiDetox and Twitter dataset to mitigate bias in Hope speech detection.
---------------------------------------------------------------
Reference 49: In tabe 5, page-13
classified under: Cognitive -> External -> ID -> IR -> Thesauri

Q6)Why External and IR type?
[49] present an indepth analysis of Web search queries for controversial topics, focusing on query sentiment. Hence the bias source should be Querying instead of External.
When [33] is Querying because it presents text-based image retrieval for complex natural language queries by using a semantic parser, why [49] is not under Querying?

Since the paper discuss about the sentiment analysis using the SentiWordNet thesaurus, the type of AI system is NLP, not IR. Even though, it focuses on query, it detects sentiment in the query.

Hence it should be:
Cognitive -> Querying -> ID -> NLP -> Thesauri
---------------------------------------------------------------
BASED ON THE ABOVE OBSERVATION OF (SAMPLE) 3 PAPERS [30,36,49]
There is ambiguity in how the above were categorized in Table 5
under different sub categories.
---------------------------------------------------------------

page 10.
Line 6-20 repeat the section 5.1 description in section 5.2. which is not required.
Instead introduction about 5.2 can be replaced.
------------------------------------------------------------

Under Section 5:Analysis of results
5.1. Bias in AI systems
--->5.1.1. Statistical bias
--->5.1.2. Cultural bias
--->5.1.3. Cognitive bias

5.2. Bias Impact
The bias impact such as population bias, content, behavioral,temporal, data quality bias are discussed separately.
Q7)
Identifying, mitigating those biases are discussed separately with NO correlation to the biases discussed in table 5.page 13.

The classification of biases are based solely on paper Olteneau et al., who described bias for social data. Should have considered/explored biases specific for semantic web (SW).
------------------------------------------------------------

Page 5: Line 30
The AI system types in survey: IR, ML, DL, RS, NLP, IA.

The ambiguity in selecting the above as AI system types.
IR,RS are application systems not technology.
ML/DL are technologies, not a system.
ML/DL are technologies that can be used in IR,RS,NLP and any AI-based systems.

IR, RS are subset of NLP.
ML,DL are applied in NLP.

No clear distinction /clarity in classification of AI types.

This article proposes a systematic approach to survey literature addressing bias with Semantic Web technologies. From a top-down perspective, 24 resulting papers are categorized into pre-existing bias typologies.

The overall idea as such is interesting and the survey methodology chosen is highly adequate. However, there are a number of questions and issues, which I detail per section prior to an overall evaluation.

The first section to define Semantic Web technologies rephrases it to conceptual semantics, however, without providing a clear definition of what is meant by “conceptual semantics”. I think this should be included. It is also questionable in how far the SKOS recommendation represents an excellent source for an overall definition of SW technologies - and it should link to the W3C page and not to the Manchester page. This general mix of SW technologies and semantics is also problematic in the remainder of the paper. In addition, a thesaurus relates to organizing a controlled vocabulary, whereas a lexical database, such as WordNet as the title of the paper suggests, organizes general language.

In reference to the survey methodology, it would be useful to more clearly define a research question, because currently the section switches between SW technologies and general semantics. The remainder of the method section raises a number of questions:
- Are the keyword presented in Table 1 all keywords utilized for the search? Some of them are marked with a star without any indication of what this means. This seems a very small number of keywords and keyword combinations for this very broad topic.
How many people were involved in the content-based filtering/screening and how many judged a single paper?
Why do papers need to address cognitive bias? Could it not be another type of bias?
- What is the reason of preferring Scopus over all other databases in the search strategy?
- How where the factors "most complete results" and "most relevant publication venue" specified that could bring the number of papers from 399 to 81?
- Snowballing only resulted in including 5 more papers? Is there any reason this number is so low?

The disparity between semantics and Semantic Web technology becomes particularly evident in the description of the different dimensions of "conceptual semantic tasks". Sentiment analysis and bias mitigation per se cannot be considered conceptual semantic tasks - maybe clarify that these are described in combination with an additional usage of SW technologies? Right after the definition of types of biases into a very high-level categorization, two additional types of bias are mentioned, i.e., functional and external biases, that are not presented in the typology. I think the typology should reflect on all types of biases utilized in the survey.

The section denominated bias impact seems to refer to problems of biases in user-generated data and how bias can result thereof. However, this is not explicitly stated. I would have expected a section along the lines of how bias impacts users. For instance, Crawford (Kate Crawford. 2017. The Trouble with Bias. In Conference on Neural Information Processing Systems (NIPS) – Keynote, Long Beach, USA.) provided an interesting classification of the impact of bias in terms of harms that could result. The category "population bias" seems to go into this direction, while the others do not - it would be good to clarify what you mean by "bias impact".

Statistical - Bias at source:
From the description of the collaborative recommender systems approach, it should be clarified where there is potential bias, i.e., the method of procuring negative samples, and how this is being addressed by the cited approach. Since both collaborative RS approaches utilize Knowledge Graphs (KGs), it is unclear to me why one is classified as Collaborative RS and the other one as Knowledge-based Collaborative RS, since both seem to fall into the latter category. The third RS category for bias at source is Hybrid RS. "Hybrid" only makes sense when the two approaches being brought together are explained, i.e., a combination of content-based and collaborative information, which is not explicitly stated.

Statistical- Bias at collection:
In this section the first category is again called Hybrid RS, however, this time the "hybrid" comes from combining a content-based RS with a KG, which is different from the other Hybrid RS but similar to a Knowledge-based RS. These categories could benefit from some refinement.

Statistical- Data pre-processing:
While the utilization of KGs can be useful for procuring more detailed image captions or facilitate question-answering over image captioning datasets, it should be explicitly stated how this approach addresses which type of bias. OOV completion can hardly be considered annotation or pre-processing. Regarding the sentiment analysis, in how far does this approach utilize/address/relate to Semantic Web technologies or only structured data? In how far could dictionary entries be considered concepts?

Cultural - Bias at source:
For the Web IR task it would be useful to indicate the type of "semantics" that has been utilized to address bias. Otherwise, in the section on cultural bias and especially cognitive bias, the types of bias and how SW technologies have been utilized to address it is more clear.

Bias impact:
While the overview in Table 4 is interesting, the corresponding resources should be mentioned where the approach is described, since, e.g. FrameNet, is mentioned for the first time in this table. The ordering of the following sections should be the same as in Table 4. To me the rationale for leaving out approaches in the table but then including them in the subsections of this section is not entirely clear (e.g. [29]) and should be explained. Additionally, the section seems to reiterate the already given descriptions of the approaches with some more details without much additional insights into the impact of bias.

In my view, a section of the impact on bias should not merely take a technological perspective considering the impact on AI systems, but rather a larger perspective on the impact on individuals, groups and the society at large. This type of discussion seems to be entirely missing in this survey.

Discussion:
The claim that the previous sections show how successful the use of SW technologies in addressing bias are would require a more detailed analysis of actual performances of individual approaches. Instead, the sections provide a list of different approaches where SW technologies have been utilized and the corresponding findings equally show where and how SW technologies have been utilized irrespective of the overall performance or other type of success. Other than that most of the discussion section seems to represent yet another (third) classification of the same 24 papers. The only insight offered from the summary of papers in different typologies is that evaluation methods are not homogeneous and and additional note on the potential bias in semantic resources is offered. This is one of the weakest points of the paper and should be considerably extended. Why is it interesting to read all these summaries of papers and what can the reader really learn from it (in addition to SW technologies are used at point x in an AI approach)? What are the main insights and challenges?

OVERALL EVALUATION:
There is a very abstract typology (statistical, cultural, cognitive bias) presented in a dedicated section that is later refined and utilized to categorize summaries of papers in the result set. I suggest providing one typology with all categories that is used consistently. Then there is another typology on bias impact that again is utilized to categorize the same set of papers. Finally, in the discussion a third typology is added to address different aspects and classify the summary of the same set of papers.

At the end of these many, partially reiterated summaries of analyzed papers, the actual insights, challenges and conclusions are rather weak, coming down to yet another typology and a discussion of evaluation methods. While it might be interesting for readers to learn where and how SW technologies have been applied in AI systems, there should be more insights than that, e.g. Where could they be applied and have not yet? Where do you expect which type of SW technology to be most effective? What else have you learned from reading all these papers? From a personal perspective and for an increased readability, I'd strongly recommend simplifying and reducing the number of typologies applied to summaries of 24 papers and change the summaries into more critical, interesting discussions.

Minor comments:
Reference [19] links to Manchester University and not to the World Wide Web consortium
3.12 bias manifest => manifests
3.31 List of keywords include => included
4.13-14 originate from
4.20 Usually first names and initials, such as R., are not included in references
4.43 referred as => referred to as
5.44 affect to different systems => affect different systems
5.1. covered by in this survey => omit "by"
6.31 their introduction has a significant change in the model parameters => what does "has" mean here? affects, causes, requires?
8.17ff "In a ML clustering analysis to provide a novel action conceptualisation" => this and the following sentences are not grammatically intact
8.11 an dataset => a dataset
8.20 from the linked open data => either with "cloud" or without "the"
9.8. relying of => relying on
10.9. those one building one => ones
10.7. Koduri et al. [35], => this is the first time that the names are utilized for references - please follow the SW guidelines here and be consistent
10.28 is based a => on a
16.17 fall even back to this problem => ???