3.1 IDP

The IDP (Imperative Declarative Programming) system (De Cat et al. Reference De Cat, Bogaerts, Bruynooghe, Janssens, Denecker, Kifer and Liu2018) is an implementation of the KBP. The knowledge in the KB is represented in a rich extension of First Order Logic (FOL), called FO( $\cdot$ ) (pronounced FO-dot). It extends FOL with types, aggregates, inductive definitions and more. FO( $\cdot$ ) is an expressive and flexible knowledge representation language, capable of modeling complex domains. The knowledge in a KB is structured in three kinds of blocks: vocabularies, structures and theories.

A vocabulary specifies a set of symbols. A symbol is either a type, predicate, or a function. A type is either a standard type such as the set of real numbers $\unicode{x211D}$ , or the name of an application-specific type, such as Adhesive. A predicate symbol either expresses a Boolean or a relation on one or more types, such as BondSealing() or Available(Adhesive). A function symbol represents a function from the Cartesian product $T_1 \times \ldots \times T_n$ of a number of types to a type $T_{n+1}$ . For example, the function $\mathit{BondStrength}: \mathit{Adhesive} \rightarrow \mathbb{R}$ maps each adhesive on its bond strength.

A (partial) structure specifies an interpretation for some of the symbols of a given vocabulary. A structure is total if it specifies an interpretation for each symbol of the vocabulary.

A theory contains a set of logical formulae in FO( $\cdot$ ).

By itself, the KB cannot be executed: it is merely a “bag of knowledge,” without information on how it should be used. The latest version of the IDP system, IDP-Z3 (Carbonnelle et al. Reference Carbonnelle, Vandevelde, Vennekens and Denecker2022), supports many different inference tasks that can be applied to this knowledge. We will briefly go over the inference tasks that are relevant: propagation, model expansion, optimization and explanation. Given a partial interpretation $\mathcal{I}$ for the vocabulary of a theory $\mathit{T}$ , propagation derives the consequences of $\mathcal{I}$ according to $\mathit{T}$ , resulting in a more precise partial interpretation $\mathcal{I'}$ . Model expansion extends a partial structure $\mathcal{I}$ to a complete interpretation I that satisfies the theory T ( $I \models T$ ). Optimization is similar to model expansion, but looks for the model with the lowest/highest value for a given term. Finally, explanation will, given a structure $\mathcal{I}$ which does not satisfy the theory T ( $\mathcal{I} \not\models T$ ), find minimal subsets of the interpretations in $\mathcal{I}$ which together explain why $\mathcal{I}$ does not satisfy the theory.

3.2 Interactive consultant

The Interactive Consultant (Carbonnelle et al. Reference Carbonnelle, Aerts, Deryck, Vennekens and Denecker2019) is a graphical user interface for IDP-Z3, which aims at facilitating interaction between a user and the system. It is a generic interface, in the sense that it is capable of generating a view for any syntactically correct KB. In short, each symbol of the KB is represented using a symbol tile, which allows users to set or inspect that symbol’s value. In this way, the GUI represents a partial structure to which a user can add and remove values. Each time a value is added, removed or modified, IDP’s propagation is performed and the interface is updated: symbols for which the value was propagated are updated accordingly, and for the other symbols the values that are no longer possible are removed. In this way, a user is guided towards a correct solution: they cannot enter a value that would make the partial structure represented by the current state of the GUI inconsistent with the theory.

At any point in time the user can ask for an explanation of a value that was derived by the system, for example, when the user does not understand it or agree with it. The system will then respond with the relevant formulas and user-made assignments that lead to the derived value. In this sense, the tool is explainable, leading to more trust in the system.

A similar functionality is in place for the rare cases in which a user manages to reach an inconsistent state, that is, a set of assignments that can no longer be extended to a solution. While the IC removes values that have become impossible, it cannot do so for variables belonging to unbounded integer or real domains. For example, it is possible to input “Min Temperature = 20” followed by “Max Temperature = 10”, as both of these variables can have any value between $-\infty$ to $+\infty$ . In the case of an inconsistency, the interface alerts the user and explains why no solutions are possible by showing the relevant design choices and laws.

The Interactive Consultant interface has already successfully been used in multiple applications in the past (Aerts et al. Reference Aerts, Deryck and Vennekens2022; Deryck et al. Reference Deryck, Devriendt, Marynissen and Vennekens2019).

4.1 Knowledge acquisition

The creation of the knowledge base is an important element in the development process of knowledge-based tools. It requires performing knowledge acquisition, which is traditionally the most difficult step, as the knowledge about the problem domain needs to be extracted from the domain expert to be formalized by the knowledge engineer. While knowledge acquisition comes in many shapes and forms, we applied the Knowledge Articulation method (Deryck and Vennekens Reference Deryck, Vennekens, Andersen, Andersen, Brunoe, Larsen, Nielsen, Napoleone and Kjeldgaard2022). The central principle of this method is to formalize knowledge in a common notation for both domain expert and knowledge engineer, so that both sides actively participate in the formalization process.

We started by organizing three knowledge articulation workshops, each lasting between three and 4 h. Each of these workshops was held with a group of domain experts. While typically a single domain expert would suffice for knowledge extraction, having a group present can help as an initial form of knowledge validation, as the experts discuss their personal way of working amongst themselves, before coming to a consensus. For the common notation we used Constraint Decision Model and Notation (cDMN) (Vandevelde et al. Reference Vandevelde, Aerts and Vennekens2021), an extension of the Decision Model and Notation (DMN) standard (Object Modelling Group 2021). DMN is a user-friendly, intuitive notation for simple decision logic. Its main component is the decision table, which allows to define the value of a number of “output variables” in terms of a number of “input variables.” Decision tables are structured together in a Decision Requirements Diagram (DRD), which is a graph that provides an overview of the total decision model by showing the connections between input variables (ovals) and decisions (rectangles). cDMN aims to increase the expressiveness of DMN (e.g. by adding constraints and quantification) while maintaining this user-friendliness.

The first workshop consisted of identifying all relevant adhesive selection parameters and using them to create an initial DRD, of which a fragment is shown in Figure 1. It is structured in a bottom-to-top way, similar to how the experts would reason: they start by calculating the thermal expansions, and then work their way up to the calculation of the maximum stress.

Fig. 1. Snippet of created DRD.

During subsequent workshops, the rest of the model was fleshed out. This consists of decision tables and constraint tables. An example of such a decision table can be found in Figure 2a. In such a table, the “inputs” (in green, left) define the “outputs” (in light blue, right). Each row represents a decision rule, which fires if the values of the input variables match the values listed in the row. If a row fires, the value of the output is set accordingly. For example, if $\mathit{Support} = \mathit{fixed}$ , then the MinElongation is calculated as $\mathit{deltaLength} / \mathit{BondThickness}$ . The (U)nique hit policy of this table, indicated in the top left, means that the different rows must be mutually exclusive.

Fig. 2. Example cDMN tables.

Figure 2b shows a constraint table (denoted by the E* in the top-left corner). In such a table, the output specifies a constraint that must hold if the input is satisfied. In other words, this table states that if the bond strength is known, then Max Stress should be higher than a minimum value. This differs from decision tables in that it does not define a specific value, but rather constrains its possible values.

After these three initial workshops, the cDMN model was converted into an FO( $\cdot$ ) KB using the cDMN conversion tool. Since then, multiple one-on-one workshops were held between the knowledge engineer (first author) and the primary domain expert (second author) to further fine-tune the KB. Among other things, this included adding a list of adhesives, substrates, and their relevant parameter values, and further validating the knowledge. In total, the current version of the KB contains information on 55 adhesives and 31 substrates. For the adhesives, the KB contains 21 adhesive parameters, such as temperature resistances, strength and maximum elongation. Similarly, it contains 11 parameters for the substrates, such as their water absorption and their solvent resistance. These parameters are a mix of discrete and continuous: in total, 15 are continuous, and 17 are discrete.

4.2 Unknown adhesive parameters

One of the main challenges in the formalization of the KB was handling unknown adhesive data. Indeed, often an adhesive’s data sheet does not list all of its properties. This raises the question of how the tool should deal with this: should the adhesive be excluded, or should it simply ignore the constraints that mention unknown properties? Together with the experts we agreed on a third approach, in which we first look at the adhesive’s family. Each adhesive belongs to one of 18 families, for which often some indicative parameter values are known. Whenever an adhesive’s parameter is unknown, we use its family’s value as an approximation. If the family’s value is also unknown, then the constraint is ignored. This best corresponds to how the experts typically work.

This way of reasoning is formalized in the KB. For example, the constraint that an adhesive should have a minimum required bonding strength is written as follows:

\begin{equation*} \begin{aligned} \forall p \in \mathit{param}: \mathit{Known}(\mathit{p}) \Leftrightarrow & (\mathit{KnownAdhesive}(\mathit{p}) \lor \mathit{KnownFamily}(\mathit{p}))\\ \end{aligned}\end{equation*}

\begin{equation*} \begin{aligned} \mathit{KnownAdhesive}(\mathit{strength}) \Rightarrow \mathit{BondStrength} =& \mathit{StrengthAdhesive}(\mathit{Adhesive}).\\ \neg\mathit{KnownAdhesive}(\mathit{strength}) \Rightarrow \mathit{BondStrength} =& \mathit{StrengthFamily}(\mathit{Family(Adhesive)}).\\ \mathit{Known}(\mathit{strength}) \Rightarrow \mathit{BondStrength} \geq & \mathit{MinBondStrength}. \end{aligned}\end{equation*}

with StrengthAdhesive and StrengthFamily representing respectively the specific adhesive’s and its family’s bonding strength. This approach is used for all 21 adhesive parameters.

One caveat to this approach is that IDP-Z3 currently does not support partial functions, that is, all functions must be totally defined. To overcome this, we assign the value $-$ 1000 to unknown parameter values, and define that the value is only known if it is different from this number. We chose $-$ 1000 as there is no adhesive parameter for which it is a realistic value.

\begin{equation*} \begin{aligned} \forall p \in \mathit{param}: \mathit{KnownAdh(p)} \Leftrightarrow \mathit{StrengthAdhesive(Adhesive)} \neq -1000.\\ \forall p \in \mathit{param}: \mathit{KnownFam(p)} \Leftrightarrow \mathit{StrengthFamily(Adhesive)} \neq -1000.\\ \end{aligned}\end{equation*}

4.3 Interface

A crucial requirement of this application is the ability to interactively explore the search space. To this end, our tool integrates the Interactive Consultant to facilitate interaction with the KB. This interface makes use of several functionalities of the IDP system to make interactive exploration possible: the propagation inference algorithm is used to show the consequences of each choice, the explain inference is used to help the user understand why certain propagations were made, the optimize inference is used to compute the best adhesive that matches all of the choices made so far.

When using the interface, the user fills in symbol tiles, each representing a different symbol of the KB, and the system each time computes the consequences. For example, Figure 3a shows a segment of the interface in which a user set a maximum application temperature of $38^\circ$ C as a requirement. To make it easier to navigate the symbol tiles, they are all divided in five categories: Performance, Production, Bond, Substrate A and Substrate B. In the top-right of the interface, the number of adhesives that remain feasible is shown: for example, after setting the temperature constraint, that drops from 55 to 12, as shown in Figure 3b.

Fig. 3. Screenshots of the interface.

The tool is also capable of generating two types of explanations. Firstly, if the user does not understand why a certain value was propagated, they can click on that value to receive a clarification, as demonstrated in Figure 3c. Secondly, if the user manages to reach an inconsistent state, the tool will try to help resolving the issue by listing what is causing it. For example, Figure 3d shows an inconsistency in which a substrate is selected that cannot handle the required operating temperature.

Besides generating a list of all the adhesives that meet certain requirements, the tool can also find the optimal adhesive according to a specific criterion, such as lowest price or highest strength.

6.1 Methodology

In our study, we held interviews with four members of the FM JML, who each possess knowledge on adhesives but have varying degrees of involvement in adhesive selection. Two interviewees are adhesive experts that often perform adhesive selection. The other two do not perform adhesive selection as part of their job, but are knowledgeable on glues in general. We included such “non-experts” to explore whether the tool is capable of making adhesive selection more accessible. We held an online one-on-one session with each of the interviewees. First the interviewee was asked to perform adhesive selection using the tool, and then we conducted a semi-structured interview to gather their opinions.

Adhesive selection. Two real-life bonding cases were selected by the second author to be used as test cases for the study: both are cases that the JML lab has received from companies in the past. In the first one, a plastic door needs to be glued to the body of an industrial harvester. Originally, the process of finding the correct glue took weeks, as the requirements are fairly tight. Moreover, the specification contains an inconsistency in which a higher temperature is required than allowed by the substrate, which is the same inconsistency as shown in Figure 3d. The second case details a join between a plastic component and an aluminum body, and is less challenging.

Both cases were presented to the participants as a short description of the gluing operation, together with a table containing the actual requirements. For the first example, the table lists requirements such as “Material A = Virgin ABS,” “Gap filling of min 1 mm,” “Application between $15^\circ$ C and $35^\circ$ C,” etc. We have taken care here to specify requirements in the same terminology as the tool. In total, the use cases consist of nine and seven requirements respectively.

Before letting the experts work on the cases, we also gave a brief introduction on how to use the tool. We explained how to enter information, where they could find the list of possible adhesives and how to see that the tool was performing calculations.

We observed each tester during the selection process, to gather information on how they interacted with the tool. To better understand their thought process, the interviewees were encouraged to talk to themselves about their process as a way to elicitate their thoughts. We did not intervene while they were working out these use cases, even when errors were made, and only made suggestions on what to do if they got stuck or an unexpected bug popped up.

Interviews. Right after the selection, we held an interview with each participant. Because our goal is to explore the opinions of the JML members, we opted for a semi-structured interview set-up, for which we prepared four main questions to serve as a general guide:

1. 1. Do you see a role for the tool in your job?

2. 2. Do you feel that you understand what the tool does when you use it?

3. 3. What was your experience working with the tool?

4. 4. How would you compare our tool to the ones that you are used to working with?

In addition to these main questions, we asked additional questions to zoom in on specific aspects of the answers given by the participants. The interviews were led by the first author, with additional support from the second author. They were audio-recorded and transcribed, so that they could be analyzed thoroughly.

After transcribing the recordings, we followed the guidelines of Richards and Hemphill (Reference Richards and Hemphill2018) and performed open coding followed by axial coding (Corbin and Strauss Reference Corbin and Strauss1990). Here, the goal is to identify various codes that pop up during the interviews, and then further group them into several main categories. These two steps were also performed separately by an external member of the research group, after which the results were compared and adapted to mitigate biases (consensus coding).

6.2 Results

Based on the interview transcripts, we identified 26 codes in total. Table 1 shows an overview of the interview statistics. The last column of this table shows the Code distribution, calculated as the cumulative percentage of codes discovered after each interview. This is an important parameter that indicates whether data saturation is reached, that is, the point at which the same themes keep recurring and new interviews would not yield new results. According to Guest et al. (Reference Guest, Namey and Chen2020), this point is reached when the difference in code distribution between the current and previous interview is $\leq 5\%$ (i.e. less than 5% of new information was found). As the information threshold between the third and fourth interview is 4%, we conclude that we have reached data saturation. A more detailed table showing the codes per interview is included in the Appendix.

Table 1. Interview statistics

We sub-divided the codes into five main themes: Knowledge, Interactivity, Expert, Interface and Explainability. These themes, together with their codes, are visualized in the graph in Figure 4. We will now briefly go over each theme and highlight the most important findings.

Fig. 4. Graph showing connections between the interview themes and their codes.

6.3 Limitations of the study

As with any study, ours is not without its limitations. The main limitation of this study is our low number of interviewees. However, as pointed out in Section 6.2, we do reach data saturation when looking at the code distribution of our interviews. Therefore, we feel that the low number of interviewees does not have a major impact.

Another limitation is that all interviewees are from the same organization and would therefore have a “common” approach to selecting glues, while external experts might have a different focus that could result in additional feedback. However, as the tool has specifically been developed for use within Flanders Make JML, we believe that this feedback would be less relevant.

During the testing of the tool, we continually observed the interviewees to study their interaction with the tool. Because of this, the users might have felt pressured to work in a more “efficient” way, as a form of the Hawthorne effect (McCarney et al. Reference McCarney, Warner, Iliffe, Van Haselen, Griffin and Fisher2007). In fact, one participant said so explicitly, when talking about their experience with the inconsistency window:

Because we were observing, we might have inhibited the users to truly “play around” with the tool and test it to their heart’s content.