Enhancing Awareness of Industrial Robots in Collaborative Manufacturing

Tracking #: 3255-4469

Alessandro Umbrico
Amedeo Cesta
Andrea Orlandini

Responsible editor: 
Guest Editors SW for Industrial Engineering 2022

Submission type: 
Full Paper
The diffusion of Human-Robot Collaborative cells is prevented by several barriers. Classical control approaches seem not yet fully suitable for facing the variability conveyed by the presence of human operators beside robots. The capabilities of representing heterogeneous knowledge representation and performing abstract reasoning are crucial to enhance the flexibility of control solutions. To this aim, the ontology SOHO (Sharework Ontology for Human Robot Collaboration) has been specifically designed for representing Human-Robot Collaboration scenarios, following a context-based approach. This work brings several contributions. This paper proposes an extension of SOHO to better characterize behavioral constraints of collaborative tasks. Furthermore, this work shows a knowledge extraction procedure designed to automatize the synthesis of Artificial Intelligence plan-based controllers for realizing a flexible coordination of human and robot behaviors in collaborative tasks. The generality of the ontological model and the developed representation capabilities as well as the validity of the synthesized planning domains are evaluated on a number of realistic industrial scenarios where collaborative robots are actually deployed.
Full PDF Version: 

Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 14/Oct/2022
Minor Revision
Review Comment:

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data, (B) whether the provided resources appear to be complete for replication of experiments, and if not, why, (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.

All previous comments have been properly addressed except: In Figure 7, 9, 11, 13, 15 the texts are still too small and unreadable.

Review #2
By Alessandro Mosca submitted on 21/Oct/2022
Review Comment:

The paper has been significantly rewritten and it now includes around 25 additional pages with respect to his first version, including details and explanations I was looking for in my previous review.
The original title of the paper, where the terms "ontology-based reasoning" were central, has been modified in such a way that those terms completely disappeared. However, I think that the abstract, which is exactly the same as in the first version of the paper, still provide a good, concise, introduction to the main content of the paper. I've also noticed the changes in the title have been implemented to answer explicit suggestions by Reviewer 3. This to say that I'm perfectly fine with the new title and that I also think it better reflects the actual content of the paper than the previous one.

The overall introduction of the SOHO ontology has been substantially modified (see sections 3.1 and 3.2): more details on the definition of the central ontological concepts have been introduced, which have a significant impact on the overall understandability of the proposed solution, in my opinion.
Here below, I list a few minor questions/doubts I ended up with while reading about the sections above:

- Since the formal specification of the SOHO ontology is provided by using a description logic notation, I don't see the reasons to introduce a first-order logic formula on page 13 and 17, whose constitutive elements are SOHO concepts and properties. Is there any special motivation behind that choice? I'm fine with the fact that they are "inference rules" but I wasn't able to find in the paper:
• how these rules formally interact with the TBox and ABox of SOHO and, more in general, how they relate with the content of the "Automated Synthesis of Plan-based Control Models" section;
• which language has been identified to represent these rules (has this language been introduced in [95]?);
• whether this formal language is compatible with that use for the "synchronisation rules" introduced in section 4.1;
• how these rules are executed.

- I don't understand the reason for the footnote on p. 18, which refers to formula #(17). Cardinality constraints/restrictions can be expressed in description logic, indeed (see "Qualified number restriction"). Which is the reason to not use these specific DL constructs and, rather, introduce the footnote? Do the authors have a specific target OWL profile in mind?
In which OWL profile SOHO falls? This would give an idea of the complexity of reasoning with it.

Section 4, pp. 19-32, is almost completely new.
My overall feeling is that this new version is sufficiently detailed and that the reader can now find how exactly specific patterns, which are described at the ontological level, play a role in the generation of the timeline-based task planning model.
The knowledge extraction procedures and the mechanisms behind the specification of timeline-based planning models, which were puzzling me in the first version the paper because of a lack of details and description, get in this second version of the contribution a completely new, more central status. I am fully satisfied by that by the way the authors described the interaction between these procedures and mechanisms with the ontology-base formal representation of knowledge and facts relying on SOHO.

I very much appreciate the fact that the authors replied to all my comments on the first version of their contribution and implemented a few of the changes I suggested there, as it appear clearly by reading the response letter accompanying the new version of the paper.
I'm in favour of accepting this new version of the paper for publication in the Semantic Web Journal, modulo some minor adjustments that refer to my comments above on the presentation of the SOHO ontology.

Review #3
By Daniel Beßler submitted on 31/Oct/2022
Major Revision
Review Comment:

The manuscript presents an extension of the SOHO ontology, and a method used to construct a "decomposition graph" from the ontology that decomposes a goal into functions of agents that can be used to achieve the goal. This connects two previous works of the authors (SOHO ontology, and a planning formalism). The contributions have been applied on a series of different industrial use-cases. Overall, the paper is widely well written and interesting to read.

I was a reviewer of the first version of the paper too. The authors have accounted for most of my concerns. I will concentrate my review on concerns which I see not fully fixed in the new version, and additional concerns arising from the huge amount of new material.

The new revision of the paper significantly improves regarding the discussion of related work. Overall, the paper has improved a lot over the last revision, and is interesting to read. But the authors have added a lot of new material which hasn't been under review in the first iteration. Judging from the number of concerns raised below, I think, a major revision is still needed.

Regarding Section 3

The paper now includes much more details about the SOHO ontology. And it seems all concepts that appear in axioms are introduced at some point. However, the paper also got substantially longer from 21 to 47 pages, which I think is a bit excessive. In case shortening is needed, I would suggest to only summarize SOHO, and to only introduce concepts that are later actually used.

- I think it is still not well presented what the starting point is, and what the extension is that is proposed in this work. The word "extension" in fact only appears in the abstract and conclusion, but nowhere in between. In the outline on page 3 it is stated that Section 3 presents "complete and refined definitions" of concepts in SOHO. So what is it now? A refinement, or an extension? Section 3 appears to mix extensions, old and refined definitions without making explicit what is what. I would strongly suggest discussing novel contributions separately from providing a summary about previous work.

- Furthermore, I am still a bit unsatisfied about the confusion DUL:Task vs. your notion of task. It appears you have chosen not to use DUL:Task, and that you rather use a taxonomy underneath DUL:Action. Since you define ComplexTask as a type of description, it might be good to elaborate about the distinction to DUL:Task to avoid confusion.

- Figure 3 seems to entail that DUL:Action is a type of DUL:Object (because ProductionMethod is a DUL:Method and therefore also DUL:Object). This would be inconsistent as Action and Object are disjoint in DUL.

- It is not clear to me how you build upon the CORA ontology. On page 8 you state that it relies on the SUMO ontology. How do you import it into a framework that uses DOLCE/DUL? Could you add more details about what parts of CORA you import for what reason, and how you do that given it is not aligned with DUL?

- eq 3,7,16 are not given in DL notation, but presumably rather FOL (but this is not explicitly mentioned). The equations lack explicit quantification which I suggest to add. Could the authors add more details about what formalism is used? I would assume something like SWRL? I further wonder about requirements for these rules to do something useful. It seems they rely on ABox data. e.g. equation (3) relies on the triples hasQuality(o,p) and observes(s,p), i.e. the quality of o must be exactly the one that can be observed by s. However, every object has its own qualities. So let's say we have n objects hosting a quality of a certain type that can be observed by s, then you would need to add n “observes” assertions to the knowledge base to make this rule work. Could you add some details about how you populate the ABox, and an explanation about why the rules are not operating on axioms but rather on assertions?

- eq 4,5,15: I am uncertain about whether the axioms have intended semantics. e.g. in equation 4 you state that a Cobot has at least one quality of type (Autonomy OR Capability OR RobotProperty). Isn't rather intended to state that each Cobot has at least one quality of each Autonomy, Capability, and RobotProperty? The same question for the other two listed equations.

- sec 3.2: could you elaborate about how you classify human properties such as ergonomics and mental status, and how these concepts are actually used? How is e.g. ergonomics further qualified or quantified? Also these concepts seem not to be used anywhere in this work, i.e. it is nowhere defined how a robot shall vary its behavior based on the properties you list. Finally I think it would help a lot to provide an example for illustration.

- minor: the text about constituency on p.15.41-46 is widely copied from DUL.owl which might violate their rights if not properly displayed as citation, or rephrased in your own words with a reference to DUL. This appeared to me because I know DUL well, but I haven't checked if there are other places where sentences were copied from other resources without rephrasing.

- minor: eq 11 implies InteractionModality is defined in DUL, but it isn't.

- minor: p.14.13-14: no text is allowed outside of column boundaries.

- minor: there seems to be a mistake in equations 18-21 where 18 uses the relation "constrains" while the others use "hasConstituent". I suppose it should be "constrains" in all of them?

Regarding Section 4

The paper now includes an introduction to timeline-based planning which is well written, and helpful for getting an intuition about the target representation of the "knowledge extraction method" proposed in this work. However, there are still quite a couple of shortcomings that need to be addressed, in my opinion.

- I have a concern that Section 3 includes definitions that are not used in Section 4 at all. In my opinion, Section 3 should be limited to the definitions that are required for the method proposed in Section 4.

- I also suggest writing the algorithms more compact, especially by using set notations.

- The section lacks an illustrative example early on. The examples provided at a later point are also not illustrative. The various listings (1-5) display output structures of the mechanism. However, the listings are not particularly helpful. They refer to a lot of meaningless names like “value_function_4”, “h_func”, “r_func”, etc. e.g. in Listing 1 it is unclear what this state variable is actually about as it only refers to meaningless names “value_1” to “value_n”. Why does “n” appear there anyways? That does not seem to be a concrete example at all. I was expecting to see some of the “extracted knowledge” in the examples including a reference to action types.

- p.19.34 Again, I wonder about how you create the ABox which is “compliant” with the TBox? As far as I can see this information is missing. I assume you would need to have a form of materialization method to do this automatically? How do you ensure the ABox is compliant with the TBox? This is not entirely trivial and should be discussed, in my opinion.

- algo.1: it is difficult to find text that describes particular sections of the algorithm. I suggest an enumeration in the text or so to make it easier.

- algo.1.1/4/8/15: Each of the lines initializes the SV variable. This seems wrong to me. E.g. between line 1 and 4 it is not even used! So why do you initialize it then? It is confusing. It also seems unintended that you overwrite the variable in each iteration in the first while loop. It also appears in other algorithms that it seems like you are nor correctly adding elements to sets, could that be?

- “extract_goals” was rather informally introduced and is being used e.g. in Algorithm 1. I would suggest removing these rather trivial functions and just write their definition in place in the algorithm. It should be pretty trivial, and fitting in one line. Same I would suggest for the various other trivial functions that the authors did not formally introduce.

- “hierarchy”, “controllability”, “human”, “robot” and other symbols are nowhere defined.

- algo.1: In general it is still difficult to follow what is going on. For instance, in step 1.2 the aim is to “extract agent-level behavior”. But I cannot see that anything is extracted except for the label in the “create_statevars” function. So effectively two state variables are created, one for the robot and one for the human. But there seems to be no extraction of knowledge at all that would then be represented in the timeline-based representation. For me it seems you could easily write a one or two-liner to initialize the state variables of goals and agents (instead of spanning the initialization over 8 lines). I would also call it just that “Initialization of state variables” as there seems to be no knowledge extracted at all in this step. Also the first while loop is not needed, just use set notation and you can write it in a single line how you map T to a set of state variables.

- minor: algo.1.10: it seems you bind the variable “H” in line 10. It wouldn’t work exactly because T is a set (i.e. no ordering) how you wrote it. You also access the set in line 13 using array notation. I would also suggest that the “hierarchy” yields a tuple of (T,H) to avoid this confusion. And T should not be a set.

- minor: algo:1.21: this notion was not introduced. E.g. “is” as a function. Why don’t you use standard notation? Also logical or is not defined for DL concepts. I would suggest you use notation from set semantics to check e.g. if an assertion exists in the ABox.

- minor: Figure 5 is not particularly well done. What are the inputs/outputs? The processes that take place? What is the meaning of arrows in the Figure? What is the meaning of the text in the nodes? Such a Figure should be designed as a kind of visual abstract, i.e. readable on its own. Put numbers or letters in the figure, refer in the text to these. Would also be good if your formal notation appears in the figure to also link the figure to formalization.

- minor: it would be good to include the function name of the defined function in the algorithms listed.

- p.22-23 I suggest using an itemized environment, which makes it easier to read.

A few typos, spelling mistakes etc appeared to me. Here are some I spotted in the first part of the paper, some of those could be easily detected by a spellchecker before the submission:

- p.2.14 allows->allow
- p.2.28 enhance->enhanced
- p.3.44-46 rewrite sentence, hard to parse, strike "if"
- p.6.20 wit->with
- p.6.22 collects->collect
- p.8.47 strike "a"
- p.9.33 describe->describes
- p.15.51 complext->complex

A few more general minor things:

- The Figures did not improve. My critique still holds that the figures are not very rich in terms of information content. They could be manually created to supply much more information.