2.1 Design creativity and design-by-analogy

A design is a means for changing an existing situation into a preferred one (Simon, Reference Simon1969). Creativity in design is often characterized as a process by which an agent (like a designer) uses its ability to generate outcomes that are novel and useful (Chakrabarti, Reference Chakrabarti, Raghavan and Cafeo2009). Researchers studied the influence of different design methods on creative design outcomes (Chulvi et al., Reference Chulvi, Mulet, Chakrabarti, López-Mesa and González-Cruz2012) and developed methods for enhancing creativity. Stimuli are important in innovative product design (Howard et al., Reference Howard, Dekoninck and Culley2010). Literature provides evidence that using an external stimulus compared to no stimulus can generate more ideas (Srinivasan et al., Reference Srinivasan, Song, Luo, Subburaj, Elara, Blessing and Wood2017). Systematic use of knowledge from artificial and natural domains helps designers generate many solutions and develop them into realizable and practical prototypes (Sarkar et al., Reference Sarkar and Chakrabarti2008). A detailed study to understand designers’ approach to select inspirational stimuli in the conceptual design phase was conducted where the representation of external stimuli was an essential factor in the inspiration process (Gonçalves et al., Reference Gonçalves, Cardoso and Badke-Schaub2016). “Design-by-analogy or Analogical Design is the process of developing solutions through mapping of attributes, relations, and purposes that a source problem or situation may share (or at least partially share) with an existing target solution or situation” (Goel, Reference Goel1997). Analogical reasoning is considered the core of human cognitive activities responsible for creativity in design (Goel and Shu, Reference Goel and Shu2015). Many empirical studies were conducted to understand the process and the factors influencing the effectiveness of Design-by-Analogy (Srinivasan et al., Reference Srinivasan, Chakrabarti and Lindemann2015; Helms et al., Reference Helms, Vattam and Goel2009; Song et al., Reference Song and Fu2019; Keshwani et al., Reference Keshwani and Chakrabarti2015; Keshwani, Reference Keshwani2018). Representation of a stimulus plays an important role in Design-by-Analogy (Hashemi Farzaneh et al., Reference Hashemi Farzaneh, Helms and Lindemann2015; Lenau et al., Reference Lenau, Keshwani, Chakrabarti and Ahmed-Kristensen2015) and it can influence fixation and creativity during idea generation (Moreno et al., Reference Moreno, Blessing, Yang, Hernández and Wood2016; Bonnardel, Reference Bonnardel2000; Vandevenne et al., Reference Vandevenne, Pieters and Duflou2016; Verhaegen et al., Reference Verhaegen, D’hondt, Vandevenne, Dewulf and Duflou2011).

2.2 Support for design-by-analogy

Search for analogues is a key step in the Design-by-Analogy process (Sartori et al., Reference Sartori, Pal and Chakrabarti2010) because finding relevant analogues can influence the nature of design outcomes. This paper outlines some of the commonly cited Design-by-Analogy support. Support like the Functional Model database (Nagel et al., Reference Nagel, Nagel, Stone and McAdams2010) and AskNature (Deldin et al., Reference Deldin and Schuknecht2014) use a function-based approach to identify analogues from a database. The Functional Model database has an engineering-to-biology thesaurus that maps biological terms to the functional basis of technical systems. AskNature categorizes the information of the biological functions according to the four layers of Biomimicry Taxonomy (Group, Sub-group, Function, and Strategy). Biological models are mapped into different engineering fields using “strategy”. Analogy Retriever (Han et al., Reference Han, Shi, Chen and Childs2018) uses 16 ontological relationships to describe the connections between various system entities. This supports analogical reasoning for creative ideation by solving proportional analogy problems.

On the other hand, supports like Functional Vector (Murphy et al., Reference Murphy, Fu, Otto, Yang, Jensen and Wood2014) and SEABIRD (Vandevenne et al., Reference Vandevenne, Verhaegen, Dewulf and Duflou2016b) use the vector space method to find analogues based on the semantic similarity of words. In the Functional Vector model, a query vector of functions is generated. A relevancy score of the query with a functional vocabulary is calculated using a patent database. SEABIRD method generates the Product Aspects (PA) and Organism Aspects (OA) matrices from a database of technical system documents and biological functions. Mapping between two domains is then quantified based on the values from the mathematical product of the PA and OA matrices.

A third category of support, like DANE (Vattam et al., Reference Vattam, Wiltgen, Helms, Goel and Yen2011) and IDEA-INSPIRE (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005), uses ontology-based data models. DANE uses data queries at multiple levels of abstractions in a controlled database comprising structured data models of SBF (i.e., Structure-Behavior-Function) (Goel et al., Reference Goel, Rugaber and Vattam2009). In SBF, ‘Structures’ are the constituent components, substances, and relations among them; ‘Behavior’ is the series of state changes from an input to an output state, and the transition from one state to another happens through functions. ‘Function’, therefore, is used as a behavioral abstraction. In IDEA-INSPIRE, the search strategy uses single or multiple levels of abstraction of the SAPPhIRE (State change – Action – Parts – Phenomenon – Input – oRgan - Effect) model (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005), see Section 2.3.

2.3 SAPPhIRE model of causality

The SAPPhIRE model has seven layers of abstraction, namely, State Changes, Actions, Parts, Phenomena, Inputs, oRgans, and Effects (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005). They together cover the system’s physical components, interfaces, interactions, structural context, and scientific laws governing them and can produce a rich comprehensive description of biological and technical systems. The SAPPhIRE model with an example (heat transfer from a hot body to cool surrounding air) is shown in Figure 1. The model is scalable to represent a complex system through a multi-instance-linked model (Siddharth et al., Reference Siddharth, Chakrabarti and Venkataraman2018). The SAPPhIRE model can be used for many design tasks; the model can be used in design synthesis and analysis (Chakrabarti, Reference Chakrabarti, Raghavan and Cafeo2009), a database of SAPPhIRE models can be used as a stimulus for Bio-inspired Design (Fu et al., Reference Fu, Moreno, Yang and Wood2014) and assessing novelty (Srinivasan et al., Reference Srinivasan and Chakrabarti2010; Jagtap, Reference Jagtap2018).

Figure 1. SAPPhIRE model of causality with an example of heat transfer from a hot body to cool surrounding air (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005; Chakrabarti, Reference Chakrabarti, Raghavan and Cafeo2009).

2.4 Structured representation from a natural language description

Finding many stimuli and maintaining diversity and variety of content are important requirements (Srinivasan et al., Reference Srinivasan and Chakrabarti2010, Yargin et al., Reference Yargin and Crilly2015). Databases with ontology-based models were developed to support the design creativity (Fu et al., Reference Fu, Moreno, Yang and Wood2014). Though ontology-based models are effective, they are handcrafted and limited in number. On the other hand, there is an abundant source of knowledge of biological and technical systems available in the form of technical documents and many websites where information is not associated with any specific format or data model. Since structured data is preferred in Design-by-Analogy because of computational advantages (Salvatore et al., Reference TM and Bidgoli2003), researchers proposed methods using the numerical techniques of ML and data sciences that can generate structured data or knowledge representation from information given in natural language descriptions (Jiang et al., Reference Jiang, Hu, Wood and Luo2022). Such methods are essentially information extraction methods, which involve skimming through documents and looking for occurrences of a particular class of information and the relationships among them (Russel & Norvig, Reference Russell and Norvig2010). Our ultimate motive is to determine the process for information extraction methods that can help extract information relevant to the SAPPhIRE model of causality accurately and repeatedly before it is automated. Hence, in this paper, we investigate the commonly cited approaches for information extraction and understand the underlying complete process. The references given in this section, therefore are not an exhaustive list but a representation of the dominant information extraction methods that researchers are pursuing.

A natural language processing (NLP)-based technique was developed to find causal relations between biological functions using a linguistic pattern of biologically meaningful keywords (Cheong et al., Reference Cheong and Shu2012). It progressively refines search keywords until a suitable match is found in a simple template to capture causal relations of biological functions (Shu, Reference Shu2010). A co-occurrence graph is another powerful information extraction method, which is used to identify relevant technical parameters for a certain domain by comparing thesauri automatically extracted from patents (Cascini et al., Reference Cascini and Zini2011). Semantic webs are created by extracting concepts, instances, and their relationships from a handbook of the engineering domain (Hsieh et al., Reference Hsieh, Lin, Chi, Chou and Lin2011). Rule-based techniques are also used to generate an engineering knowledge graph from a patent database with greater size and coverage (Siddharth et al., Reference Siddharth, Blessing, Wood and Luo2022). Semantic networks are considered effective for knowledge representation from vast data sources, so many such networks were developed (Chen et al. Reference Chen, Jia and Xiang2020; Han et al., Reference Han, Sarica, Shi and Luo2022). Though the large technology semantic knowledge graph, TechNet, is reported to be better than other semantic networks such as WordNet or ConceptNet for engineering applications, more study is needed with TechNet or similar networks (Han et al., Reference Han, Sarica, Shi and Luo2022; Sarica & Luo, Reference Sarica and Luo2021; Sarica et al., Reference Sarica, Han and Luo2023).

Ontology-based models can provide multiple levels of abstractions of a system and can be used to explain systematically the functioning of a system (Srinivasan et al., Reference Srinivasan, Chakrabarti and Lindemann2012). Domain ontology can be extracted using design themes (things that are of the designer’s interest) from a set of technical documents (Li and Ramani, Reference Li and Ramani2007). Therefore, ontology-based information extraction (OBIE) is another type of information extraction (Wimalasuriya and Dou, Reference Wimalasuriya and Dou2010). A two-step process using a supervised learning technique is developed where the first step identifies the occurrences of the candidate semantic entities and then identifies the ontology-specific relationships between them in the next step (Nédellec et al., Reference Nédellec, Nazarenko and Bossy2009). Text mining techniques can be used to define a domain ontology itself, which is subsequently used in knowledge extraction and visualization (Yang et al., Reference Yang, Kim, Hur, Cho, Han and Seo2018). The biologically inspired adaptive growth approach (BIAG) learns domain ontologies from engineering documents utilizing the similarity in domain ontology learning process with a tree growth (Zhang et al., Reference Zhang, Zhou, Chang and Yang2020). Due to its extensive knowledge representation, the ConceptNet semantic network with 16 generic ontological relations is also used to create an ontology-based representation (Han et al., Reference Han, Shi, Chen and Childs2018). However, how to use ConceptNet semantic network or knowledge graph for classical design ontologies such as SBF (Goel et al., Reference Goel, Rugaber and Vattam2009) or SAPPhIRE (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005) has not been reported.

Causal representations of systems play a significant role in creative design ideation (Baldussu et al., Reference Baldussu, Cascini, Rosa and Rovida2012). Hence, this research focuses on an ontology that supports causal reasoning in design, a central theme of Analogical Reasoning (Lee and Holyoak, Reference Lee and Holyoak2008; Waldmann and Hagmayer, Reference Waldmann, Hagmayer and Reisberg2013). Results show that causal models, such as SAPPhIRE or SBF are effective in design ideation (Sarkar et al., Reference Sarkar and Chakrabarti2008; Kim et al., Reference Kim and Kim2011). Hence, we pay attention to methods that automatically create representations of text using a causal ontology, such as SAPPhIRE or SBF. A support called IBID (Intelligent Bio-Inspired Design), was developed to convert natural language description into SBF-based description (Goel et al., Reference Goel, Hagopian, Zhang and Rugaber2020). IBID uses a combination of knowledge and machine learning-based approaches to extract and represent the knowledge using the SBF model. This work on IBID reported a detailed comparative study of multiple ML algorithms used to classify knowledge using Structure, Behavior and Function tags. Another research reported a four-step process for converting natural language descriptions into descriptions structured with the SAPPhIRE model (Keshwani et al., Reference Keshwani and Chakrabarti2017, Keshwani, Reference Keshwani2018). This work extracts all the sentences with potential SAPPhIRE constructs first. After that, they are split into words or collections of words. In this work, the researchers developed a classifier using the support vector machine of the ML technique to classify the SAPPhIRE label of each word or clause.

These last two forms of support were developed to create the SBF and SAPPhIRE ontology-based descriptions from natural language text, which are closely aligned with the research goal of this paper. These supports use supervised learning with hand-annotated data for training and validation. However, none of these explains how to create hand-annotated data accurately. Moreover, these supports are semi-automated with major decision-making touchpoints that are currently executed manually by specialists. These manual touchpoints are not documented. The entire process of creating a structured data model from a natural language description and its overall accuracy with design examples is not reported. Therefore, the complete process and the main factors influencing the creation of ontology-based models need to be adequately researched. In an empirical study comparing three approaches, namely keyword search using Ask Nature (handcrafted database), NLP-based approach (unstructured corpus) and one where Biologists manually perform the search in literature, it is found that though the NLP-based technique is promising, it is currently prone to errors (Willocx et al., Reference Willocx, Ayali and Duflou2020). This implies that it is important to understand the process before applying the NLP techniques.

3.1. Research question

The aim is to find the process that can serve as the basis for developing automation for creating SAPPhIRE models from information in natural language documents. Hence, the commonly used information extraction methods reported in the literature are studied. We find that ‘ML’ based methods were developed that worked along with manual steps for creating causal representations from systems descriptions in natural language. Research opportunities exist in understanding and documenting this entire process so that this understanding can act as the basis for accurately creating causal models before automating the process. Hence, this research investigates (1) understanding the entire process of extracting information and representing this information in the desired model and (2) developing an improved process. Since the SAPPhIRE model of causality can be used to provide a rich description of biological and technical systems for design analysis and synthesis, we used it to develop and test the process. This improved manual process will be automated only after it becomes accurate and repeatable.

Therefore, the research question in this research is:

“How to create an accurate and repeatable SAPPhIRE ontology-based representation to describe the working of technical systems from a natural language description?”

This main research question is further divided into the following two sub-questions:

1. 1. What is the current process of extracting information and developing SAPPhIRE representation from a given natural language description about the working of a system?

2. 2. How should the process be modified for improving information extraction and developing SAPPhIRE representation from a natural language description?

3.2. Research methodology

This research adopts an iterative and user-centric approach as shown in Figure 2, based on the design research methodology framework of Blessing and Chakrabarti (Reference Blessing and Chakrabarti2009). The cycle starts with understanding and improving the current process by addressing the gaps. Each research cycle produces a revised process by working with the users of the SAPPhIRE model. The performance of the revised process is then evaluated through user trials. This cycle is repeated multiple times until a process that delivers the target performance is reached.

Figure 2. Research cycle.

Since there are two research sub-questions involved, the results are presented in two parts. The first part summarizes the research carried out with the SAPPhIRE specialists (Bhattacharya et al., Reference Bhattacharya, Bhatt, Ranjan, Keshwani, Srinivasan and Chakrabarti2023). The first part aims to understand the current process of generating the SAPPhIRE representation. In the first part, improvement opportunities are identified and a process is proposed inspired by the cognition of the SAPPhIRE specialists. The second part of the research develops the above process further by generalizing it. Hence, the second research cycle works with novice users of the SAPPhIRE model. The reasoning rules for SAPPhIRE information extraction are developed and validated as part of this research. These rules will be useful for later development of support for automation.

4.1. Intercoder reliability study with SAPPhIRE specialists

The first step is to understand the current process of developing a SAPPhIRE model from a natural language description of the working of a system. The work for this step is carried out by working with the SAPPhIRE specialists. These specialists are researchers who have used the SAPPhIRE modeling for at least four years in their research. This work starts with an observational study by asking the four SAPPhIRE specialists to generate the SAPPhIRE representations from natural language descriptions of four different systems. The four specialists produce the SAPPhIRE-based representations based on their understanding of the SAPPhIRE modeling (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005) and interpretation of the descriptions in natural language. The data from these models are then used to calculate and report the intercoder reliability study. The procedure of this observational study is given in Appendix A. This study uses the natural language descriptions of four systems, namely, ‘Elephant Turbinate’, ‘Bombardier Beetle’, ‘Thermal Wheel’, and ‘Electric Horn’ and represented by the sample ID ‘EG1’, ‘EG2’, ‘EG3’ and ‘EG4’ respectively. The contents of these samples, given in Appendix A, are hand-curated with information taken from commonly available websites such as howstuffworks.com, asknature.org, and Wikipedia.

Table 1 shows the results of the intercoder reliability of this study. The results in Table 1 show the differences among the specialists regarding the SAPPhIRE representation of the system descriptions. The first column shows the Sample ID, as explained above. For example., The sample ID EG1 represents the case of how ‘Elephant Turbinate’ functions. Results for each Sample ID, shown in individual rows, are described in the subsequent columns. The second column in each row gives the total number of words relevant for all the seven SAPPhIRE constructs in a given sample, as identified by the four specialists. For example, for EG1, all four specialists identified 71 words (given in the second column) relevant to the SAPPhIRE constructs. The subsequent columns show the number of words and their percentage (%) out of the total number of words given in the second column, where a minimum of three and two out of four specialists had agreed. For example, for EG1 out of the 71 words, in 6 of them (i.e., 8%), at least 3 out of 4 specialists had an agreement (given in columns 3 and 4), and in 21 of them (i.e., 30%), at least 2 out of 4 specialists had an agreement (given in columns 5 and 6).

Table 1. Intercoder reliability study as per current SAPPhIRE modeling

Workshops are then conducted to understand the reasons behind the differences. In the absence of any defined process for SAPPhIRE modeling, interpretational differences played a role in interpreting the content of the natural language text among the users. Interpretation can happen at an individual word or sentence level, at multiple sentence level, or the entire document level. The text in natural language often describes the technical process at a high level and does not provide enough details. However, the current process does not assess what information is given and what is missing in the document. It was found that due to differences in the interpretation of the text, the definition of the SAPPhIRE constructs is also not applied consistently. It is also observed that different technical or scientific terminologies are used, based on the specialist’s background knowledge, to represent the same physical behavior.

The details of this user study, including the intercoder reliability analysis, are reported by Bhattacharya et al. (Reference Bhattacharya, Bhatt, Ranjan, Keshwani, Srinivasan and Chakrabarti2023).

4.2. Identifying improvement opportunities for the current process

Workshops with SAPPhIRE specialists after the first intercoder reliability study give inputs on how the consistency across the models could be improved. Inspired by the cognition of the SAPPhIRE specialists, a process called New Process (Preliminary) is proposed to extract SAPPhIRE-related information from technical documents in natural language. Another observational study is then carried out with the same SAPPhIRE specialists and the New Process (Preliminary) to check if the models’ consistency improved. In this process improvement, the first goal was to understand the information given in the context of the entire document before extracting individual information relevant to each SAPPhIRE construct. The process proposed above, therefore, has two parts:

• • Part A: This part is about Knowledge Understanding and Semantic Extraction. Here, the objective is to understand the information given in the context of the whole document, not just the sentence. A knowledge graph-based approach was recommended for this part, keeping in mind future automation. An example of a Knowledge Graph representing information relevant to SAPPhIRE from a system description document is given in Appendix B. However, generating the knowledge graph is not mandatory while manually executing the process. While manually performing this task, the agent (designer in this case) must understand the information in the context of the whole document instead of the context of the sentence at hand.

• • Part B: This part is about extracting SAPPhIRE-related information. Here, rule-based reasoning is performed by the agent to identify information that is relevant to each SAPPhIRE construct. For automation in the future, these rules will be converted into a propositional logic of an expert system.

User trials of this new process with specialists, carried out using the procedure given in Appendix A, show that this process has the potential to improve the consistency of SAPPhIRE information extracted. Intercoder reliability analysis with specialists before and after process development is given in Table 2. Description of the columns given in the explanation of Table 1 hold goods for Table 2 as well. Table 2 shows that the score (% of SAPPhIRE words/clauses where 3 out of 4 specialists agreed) improved from 3% to 61%. Analysis of the results gives the improvement opportunities that need to be looked into before generalizing the process. The sample IDs ‘EG1’, ‘EG2’, ‘EG3’ and ‘EG4’ in Table 2 are the same as in Table 1. Sample IDs ID ‘EG5’, ‘EG6’, ‘EG7’ and ‘EG8’ are new samples representing descriptions of ‘Electric Battery’, ‘Solar Heater’, ‘Mechanical Lock’ and ‘Visualizing Infrared Rays by Fish’, respectively and given in Appendix A. The details of the process improvement opportunities based on the workshop with SAPPhIRE specialists and the user trials conducted with the new process are reported in (Bhattacharya et al., Reference Bhattacharya, Bhatt, Ranjan, Keshwani, Srinivasan and Chakrabarti2023). This process developed based on the workshop with the SAPPhIRE specialists is called the New Process (Preliminary) since it needs to be improved further and is not generalized yet.

Table 2. Intercoder reliability scores WITHOUT and WITH the new process (preliminary)

It should be noted that the intercoder reliability score implies the level of repeatability in the SAPPhIRE model. A low intercoder reliability score does not necessarily imply low accuracy. The accuracy of these SAPPhIRE models was not checked individually since each of these was developed by a SAPPhIRE specialist, which was taken as a marker for accuracy.

5.1. Generalizing the process

For the second research sub-question, the New Process (Preliminary) described in Section 4.2 is taken as the starting point and developed further to make it a ‘to-be’ process for the future. As the work progresses on developing the process and generalizing it, the following items are addressed:

1. i. Extend rules for extracting the SAPPhIRE information from technical documents outside of Natural Language Processing (i.e., the parts of speech tags of the words and their syntactical relationship) to include rules based on system modeling techniques (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005; Hubka & Eder, Reference Hubka and Eder2002). With this change, the category of extracted information is ascertained to determine the most appropriate SAPPhIRE tag.

2. ii. An engineering or biological system can have more than one system interaction and currently there is no rule to identify each separately. Hence, the rules need to identify all meaningful system interactions.

3. iii. The rules should be able to differentiate a phenomenon from an action and to identify input and state changes if they are not explicit in the natural language description.

4. iv. The specialists participating in the intercoder reliability study are familiar with the concepts of the SAPPhIRE modeling. Since the novice users are not familiar with SAPPhIRE modeling, the process should guide them in creating system representation using SAPPhIRE

The first three items described above are related to the reasoning rules for SAPPhIRE information extraction given in Part B of the process. This process was studied previously with the SAPPhIRE specialists. These changes will not impact the process step but will enhance the rules it uses. However, the process or tasks associated with the last bullet item are not defined. This is about representing the extracted information that was not studied before. Therefore, learning and documenting the SAPPhIRE representation process is required first. Hence, work for this second step is carried out in two phases. In the first phase, a simple process called a New Process (Intermediate) is created and a user trial is conducted to observe user cognition. This user trial’s observations are then used to define the New Process (Final). The user trial of this first phase also demonstrates whether novice users can follow process instructions for creating representation using SAPPhIRE. Therefore, lessons from this first user trial are used to refine the final user trial at the end.

5.2 User trial procedure

Unlike user trials for the research sub-question 1, users participating in the user trial this time are unfamiliar with the SAPPhIRE modeling concept. So, before asking them to model, a knowledge session is introduced to train them on understanding systems, systems boundaries, interactions, and definitions of SAPPhIRE constructs. The steps of the User Trial procedure are given in Figure 3. The models created during the user trial are analyzed by comparing them against the reference data set (i.e., information relevant to SAPPhIRE constructs) created by SAPPhIRE specialists. The effectiveness of the process is checked by using the model score. The model score reflects the accuracy and completeness of the SAPPhIRE models created using the process. Sample documents are provided to every participant which are used as use cases during the user trial. These documents are hand-curated to cover information related to every SAPPhIRE construct. The technical information about how the system works, as given in the document, is assumed to be accurate. The intention of the user trial is to check whether a participant can identify the system interactions and build the causal chain for each interaction by identifying the parts, conditions or attributes, external input, governing scientific law associated with that interaction and the state change it produces. Action is the high-level interpretation of a State Change. Hence, it is more important to identify the State Change accurately. The construct Action is kept out of scope as of now in the user trials part of the second step. A training session is conducted to ensure everyone understands the SAPPhIRE concept and the modeling process before creating models.

Figure 3. Procedure for the user trial.

The choice of a document to be used as a use case becomes critical since the linguistic expressions in the document can be ambiguous and may create interpretational differences due to several factors, such as semantics, writing style, problem context, etc. (Halevy et al., Reference Halevy, Norvig and Pereira2009). It is found that designers generate solutions out of multiple analogies by breaking the main design problem into sub-problems. Designers most often search online sources for inspiration to solve the sub-problems (Vattam et al., Reference Vattam and Goel2013). Hence, short technical articles with content harvested from online sources are used to validate the process. These short technical articles represent the cases designers typically use during ideation and such short descriptions are about ‘how a system works’ and, therefore are causal descriptions of systems’ working. Technical documents with high school-level science knowledge are used to ensure all user trial participants have the same knowledge of the subject. The background of the participants of the user trials is shown in Sections 5.3 and 5.4.

5.3 First user trial

In the first phase, a simple process, as shown in Figure 4, is developed to create a SAPPhIRE model from a technical document. As mentioned in Section 5.1, this process is called the New Process (Intermediate). There are no elaborate guidelines for steps 2 & 4 of this process. This process is kept simple with a few steps. Illustrative examples created by specialists are shared with the participants in the user trial. As all the participants have an engineering background, the objective is to watch out how the participants carry out these two steps and identify if there is any common behavioral pattern among the participants. For step 3, the reasoning rules are used and these rules were created earlier (while working on the first step) for extracting SAPPhIRE information and updating, as per the first three items given in Section 5.1. The revised reasoning rules for information extraction are given in Appendix C.

Figure 4. Process diagram of the new process (intermediate) for the first user trial.

The first user trial was conducted at the Centre for Product Design and Manufacturing department of the Indian Institute of Science. There was a total of 7 students. Participants were from an engineering background without any significant job experience. None of them worked with the SAPPhIRE model before. The participants attended a knowledge session before creating a model. Each participant was given a single use case given in Use Case 1 of Appendix E. This use case is about working of an engineering system with two main system interactions. This use case was hand-curated and the reference SAPPhIRE model for the use case was created by a SAPPhIRE specialist.

5.4 Results from first user trial

The SAPPhIRE models created by each participant are compared against the reference SAPPhIRE model. The percentage of relevant words identified by each participant for each SAPPhIRE construct is calculated first and then their average. The results show that the average percentage of the relevant words identified by the participants is not high for any of the SAPPhIRE constructs. On closer look, it is found that the participants used systems interaction to decide the number of SAPPhIRE models. We see that there is variation among participants in identifying the number of system interactions (i.e., Phenomena). For example, two participants identified 2 Phenomena, three participants identified 3 Phenomena and two participants identified 4 Phenomena, indicating that interpretation plays a dominating role in deciding the number of SAPPhIRE models. As a result, there is variation in identifying other SAPPhIRE constructs. If a particular Phenomenon is not identified, other associated constructs are also not reported. Results improve significantly when only those two Phenomena are taken, which are common among all the participants, along with the Parts, Organ, Input and State Change associated with these two Phenomena. Figure 5 shows the percentage of the total number of relevant words identified by the participants on average for all four Phenomena and the two common Phenomena.

Figure 5. % of the total number of relevant words identified by the participants on average in the first user trial.

Based on the analysis of the results and with feedback from the participants, the following lessons from this first user trial are summarized. These are then incorporated into the final user trial procedure:

• • System interaction (i.e., Phenomena) can be the basis for the number of SAPPhIRE models. In the absence of a prescriptive guideline, individual interpretation varies in identifying system interactions, resulting in differences in the final model.

• • In the first user trial knowledge session, the participants were not clear on what to do (concept) and how to do it (procedure). These two things should be separated.

• • The knowledge session should have less presentation and more interactive discussion. Due to limited interactions during the knowledge session in the first user trial, participants did not get opportunities to clarify doubts about the SAPPhIRE concept and process guidelines.

• • The first user trial process was heavy on the use of work product templates and the purpose of these templates was unclear to the participants.

• • Except for the Organ, in general, all the words identified by the participants are relevant for the SAPPhIRE constructs, and very few words were not relevant for the modeling. This implies that when the process is applied correctly, it produces relevant results.

5.5 Final user trial

Based on the lessons learned from the first user trial, a prescriptive process was created for information extraction and representation using the SAPPhIRE model. As mentioned in Section 4.1, this process is called the New Process (Final). The process diagram of the revised process is shown in Figure 6. An illustration of the process is given in Appendix F using the example of the working of a Solenoid Valve, whose description is also given in the same Appendix.

Figure 6. Process diagram of the new process (final) for the final user trial.

The number of work product templates is reduced to one template in the final user trial. For the knowledge session, interactive discussion is emphasized over presentation. The SAPPhIRE concept is first explained, and then, the modeling process is taught before asking the participants to generate the SAPPhIRE representation.

The SAPPhIRE Modeling table to organize the extracted information is given in Appendix D.

5.6 Results of the final user trial

The following metrics are used to analyze the models created by the participants.

Approach A (Level of Agreement with the reference data set): This approach is based on the number of participants who correctly identified a SAPPhIRE construct. For each relevant word/clause in the reference data set, calculate the agreement score in terms of percentage as:

(i) $$ {\displaystyle \begin{array}{c}\%\mathrm{Agreement}=\\ {}\frac{\mathrm{No}.\mathrm{of}\ \mathrm{participants}\hskip0.35em \mathrm{who}\hskip0.35em \mathrm{identified}\ \mathrm{that}\ \mathrm{word}\hskip0.32em \mathrm{as}\hskip0.32em relevant}{\mathrm{Total}\hskip0.32em \mathrm{No}.\mathrm{of}\ \mathrm{participants}}\hskip0.24em \mathrm{x}\;100\;\end{array}} $$

Then, calculate the agreement of each SAPPhIRE construct based on all the relevant words/clauses belonging to that SAPPhIRE construct. This % Agreement score indicates, on average, how many participants identified the words/clauses determined as relevant by a specialist. The % Agreement score, therefore, implies repeatability since higher agreement indicates many participants can apply the rules consistently.

Approach B (Accuracy): This approach is based on Precision and Recall scores (Russel & Norvig, Reference Russell and Norvig2010). Here, first, calculate each participant’s Precision and Recall for each SAPPhIRE construct. Then, calculate the average Precision and Recall of all the participants. A precision value indicates how many of the identified Words/Clauses are relevant. A recall value indicates how many of the relevant words/clauses are identified by the participant.

(ii) $$ \mathrm{Precision}=\frac{\mathrm{No}.\mathrm{of}\ \mathrm{True}\ \mathrm{Positives}}{\mathrm{No}.\mathrm{of}\ \mathrm{True}\ \mathrm{Positives}+\mathrm{No}.\mathrm{of}\ \mathrm{False}\ \mathrm{Positives}} $$

(iii) $$ \mathrm{Recall}=\frac{\mathrm{No}.\mathrm{of}\ \mathrm{True}\ \mathrm{Positives}}{\mathrm{No}.\mathrm{of}\ \mathrm{True}\ \mathrm{Positives}+\mathrm{No}.\mathrm{of}\ \mathrm{False}\ \mathrm{Negatives}} $$

(iv) $$ \mathrm{F}1\;\mathrm{score}=\frac{2\ast Precision. Recall}{\left( Precision+ Recall\right)} $$

The terms True Positives, False Positives, and False Negatives used in the equations (ii) (iii), and (iv) are defined in Table 3.

Table 3. Definitions of terms true positives, false positives and false negatives

Figure 7 shows the results of the analysis of models using Approach A (equation i) and Approach B (equation ii-iv). Figure 8 shows the number of system interactions identified by the participants. Figure 9 shows % Agreement scores for use case 1 and use case 2 separately, mainly to check for any noticeable differences between them.

Figure 7. Analysis of models in the final user trial.

Figure 8. Number of system interactions identified by the participants in the final user trial.

Figure 9. % Agreement scores for use case 1 and use case 2 in the final user trial.

5.7 Discussions of results of the final user trial

The following observations are made from the final user trial:

• • The average Agreement score of 87% and the Average F1-Score of 89% are in the same order. 87.5% of participants reported two instances correctly for both use cases.

• • In general, it is found that precision is higher than the recall values. High precision means lesser numbers of irrelevant words/clauses (i.e., More True Positives and fewer False Positives). Out of 16 participants, 13 of them on average, could identify the 32 Relevant words/clauses (determined by specialists). On the other hand, there were 10 words/clauses which are not Relevant but only 1.5 participants on average marked them as Relevant. This implies that the participants can follow the process in most situations.

• • However, lower recall values indicate that a few participants did miss identifying relevant information. This could be due to the lack of clarity in the input natural language documents and the knowledge gap of some participants.

• • Except for Organ, precision values are higher than 95% for the other SAPPhIRE constructs. However, the recall values are lesser. This is primarily because some participants missed identifying one or two Phenomena and associated other SAPPhIRE constructs like the Effects, Parts, Inputs, and State Changes of the linked Phenomena (e.g., two participants missed to identify Blackbody radiation as the Phenomenon in the first use case and two participants missed to identify Volume Expansion due to heating as the Phenomenon in the second use case).

• • In both use cases, a few parts are mentioned whose causal role is not given in the document. Since these material entities represent a system object, they were already included in the reference data set. Some participants reported them as Parts, while others ignored them. Hence, a reduction in recall values for the Parts is observed.

• • Precision and recall values for the Organ are much lower than the other SAPPhIRE constructs. This could be due to a knowledge gap. It is noticed that most participants overlooked the condition for system interaction given in the document. E.g., in the case of Heat Transfer, the temperature gradient is a condition for the Phenomenon to occur. But except for one participant, the rest overlooked it. Similarly, a lightbulb emits light only when the filament temperature is high enough. Though this was mentioned in the document, a majority of the participants overlooked it. 50% of the false positive information is related to Organ. These would be investigated by improving the rules and guidelines around Organ in the next revision.

• • The observations from the final user trial are used to validate the rules, which would be useful for developing automation support later. It is found that the rules work without any changes in 93.75% of the 32 relevant (true positive) words/clauses. However, rules need to be updated in 6.25% of the 32 relevant (true positive) words/clauses. In these cases, rules did not work but participants used their domain knowledge in heat transfer to identify the relevant words for the SAPPhIRE construct. Also, 10 words/clauses are not relevant but participants falsely interpret the word/clause as a true SAPPhIRE construct (False Positive). These cases would be looked at in the future.

• • Researchers previously reported function-based reasoning as a pervasive and important basis for conceptual designs (Umeda and Tomiyama, Reference Umeda and Tomiyama1997; Mulet et al., Reference Mulet, Chakrabarti, Chulvi, López-Mesa, González, Porcar and Galán2009; Escrig et al. Reference Escrig, Chakrabarti, Chulvi, Mesa, González, Porcar and Galán2009). In this research, it is observed that identifying system interactions at the beginning and then selecting each system interaction as a Phenomenon to use as the basis for finding other SAPPhIRE constructs aligns with the cognition of designers and leads to better consistency and accuracy of information extraction. If ‘Function’ is considered as what the system is intended to do or produce (Chakrabarti, Reference Chakrabarti1998), it then points to the interaction or interactions a system has with its surroundings. This might be a major reason as to why the process led to better consistency.