1. INTRODUCTION
Reasoning about function of products is critical in product development, which has led to many approaches to functional modeling being advocated to support systematic products development. The selection of the approach deemed most appropriate for a particular reasoning need remains problematic, however, given the lack of clear guidance for informing this decision (Otto & Wood, Reference Otto and Wood2001; Ulrich & Eppinger, Reference Ulrich and Eppinger2008; Ullman, Reference Ullman2010; Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013). Therefore, this paper provides a justification and a proposed research direction for establishing a common benchmarking scheme for function representations that are developed and deployed throughout academia and practice with the ultimate goal of providing industry with practically usable functional modeling tools and concepts and a clear rationale for selecting a particular one. Despite decades of research into functional descriptions, research suggests industry has yet to incorporate functional modeling in practice in a systematic way, while still proclaiming a need to express product information beyond form (Eckert, Reference Eckert2013; Tomiyama et al., Reference Tomiyama, Van Beek, Cabrera, Komoto and D'Amelio2013; Arlitt et al., Reference Arlitt, Stone, Tumer, Chakrabarti and Lindemann2016). It should be noted that there are other representations that are similar, but are not directly associated with function modeling, that are used in industry such as p-diagrams (Telenko & Seepersad, Reference Telenko and Seepersad2010; Campean et al., Reference Campean, Henshall and Rutter2013), block diagramming (Sturges et al., Reference Sturges, O'Shaughnessy and Kilani1996; Braha & Maimon, Reference Braha and Maimon1998), or IDEF0 (Nagel et al., Reference Nagel, Perry, Stone and McAdams2009; Buede & Miller, Reference Buede and Miller2016). Another possible reason for this resistance is the lack of a canonical definition of function with each approach grounded in different conceptualizations, or the possibility of multiple distinct concepts inherent within a shared terminology.
Researchers and practitioners have proposed many different views of function in engineering design (Deng, Reference Deng2002; Crilly, Reference Crilly2010; Srinivasan et al., Reference Srinivasan, Chakrabarti and Lindemann2012; Eckert, Reference Eckert2013; Goel, Reference Goel2013; Vermaas, Reference Vermaas2013), with three recent approaches included in this Special Issue on function being the dimensional analysis conceptual modeling framework (Hossein et al., Reference Mokhtarian, Coatanea and Henri2017), critical chain models (Agyemang et al., Reference Agyemang, Turner and Linsey2017), and system state flow diagrams (Yildirim et al., Reference Yildirim, Campean and Williams2017). These concepts in turn have been used to inform the creation of many approaches for modeling information about a product's functionality. For example, several design textbooks emphasize the use of function-flow networks to capture the sequence and dependencies for the desired functionality of a product (Otto & Wood, Reference Otto and Wood2001; Ulrich & Eppinger, Reference Ulrich and Eppinger2008; Ullman, Reference Ullman2010; Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013).
Rather than develop a single, unified definition of function, we assert that each approach has its own strengths and weaknesses. Although each approach is useful and particularly well suited for different reasoning applications and domains, their transference across different domains remains a difficult proposition. Therefore, we propose a different approach to function research through the creation of a set of comparative benchmarks that can be explored with the different modeling approaches. The community may in turn use these proposed benchmarks to discern which approaches are more useful for different needs, and perhaps to discover which elements of the representations and vocabularies are most conducive for different elements of functional thinking.
The information captured within function models can be used to facilitate many different engineering activities across the entire product lifecycle, such as synthesis, analysis, exploration, visualization, explanation, and fault detection (Gero & Kannengiesser, Reference Gero, Kannengiesser and Gero2002; Goel & Bhatta, Reference Goel and Bhatta2004; Kurtoglu & Tumer, Reference Kurtoglu and Tumer2008). While modeling approaches might be defined clearly, they do not always come with clear guidance on how to represent specific models. Unfortunately, these modeling approaches and representations are perceived as easy neither to use nor to learn, with potential users remaining poorly informed with respect to what these representations can provide. Therefore, both of these assumptions represent an educational challenge for the community with the notion of function even considered as unclear (Eckert et al., Reference Eckert, Alink, Ruckpaul and Albers2011). Thus, these assumptions must be addressed with each representation and modeling approach proposed, but with common frames of reference, perhaps supported through a standard benchmarking protocol that would define common problems and common issues against which methods can be challenged.
2. THE NEED FOR BENCHMARKING
We next turn to other canonical benchmark systems that have proven useful in the cross-comparison of algorithms and methods. Despite the subject of repeated analysis in the research community, benchmarking has rarely been applied to the methods and tools developed in the engineering design community. One of the few examples of an effort to directly compare different function representations focused on additive manufacturing processes (Summers & Rosen, Reference Summers and Rosen2013). For example, software engineering may entail the use of benchmarking to compare algorithms (Dolan & Moré, Reference Dolan and Moré2002), specifically with regard to the traveling salesman problem, which involves a series of benchmark problems used for such comparisons (Peterson, Reference Peterson1990). Likewise several accepted optimization benchmark problems have been used to evaluate performance of new algorithms (Brest et al., Reference Brest, Greiner, Boskovic, Mernik and Zumer2006). Similarly, standard benchmark tests have been used in the automotive industry to consider different control strategies (Rajamani, Reference Rajamani and Ling2012).
Each of these different benchmark sets have been constructed to test new algorithms, either optimization or controls. With developing a benchmark set of problems for comparing function representations, the algorithm, or reasoning dimension, must be considered, as should the representation and the modeling of the functions, thus defining a critical distinction between the traditional approaches of benchmarking and the approach proposed here.
No protocol for benchmarking functional modeling currently exists, nor has there been any systematic comparison of the expressive power of various models been, despite the individual comparisons available in the literature of such relationships. A huge variability between individuals and in particular between different modeling approaches exists, as indicated in the experiment on functional descriptions, where different engineers were provided with a product and asked to generate a functional description (Eckert et al., Reference Eckert, Ruckpaul, Alink and Albers2012). Consequently, a systematic analysis is needed to determine how different modeling approaches compare in representational expressive power, reasoning inferencing capacity, and modeling ease of use.
3. IS FUNCTION RESEARCH SUFFICIENTLY MATURE?
Before defining a series of benchmark test cases for use in the cross-evaluation of competing function representations, we must first determine if the research field is sufficiently established to warrant such an effort. This is critical for determining both the sufficient need in a plethora of competing approaches and a sufficient population size of researchers willing to use these benchmarks as comparative tools. To this end, we first consider the field's evolution over the past five decades (Table 1). The evolution has been characterized by incremental shifts and the creation of function vocabularies (from normative function descriptions, Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013; to controlled vocabularies, Hirtz, Stone, McAdams, et al., Reference Hirtz, Stone, McAdams, Szykman and Wood2002; to physics-defined vocabularies, Sen et al., Reference Sen, Summers and Mocko2013c ). It has also involved the creation of conceptually divergent approaches for modeling interface-centered (Wang et al., Reference Wang, Qing-liang and Albers2009), component-centered (Fenves et al., Reference Fenves, Foufou, Bock and Sriram2008), and user-centered functionalities (Gaffney et al., Reference Gaffney, Maier and Fadel2007). Functional modeling currently receives a renewed interest as the greater integration of mechanical systems with electric systems and software required trade-off across systems on a functional. For example, the integrated function modeling framework (Gericke & Eisenbart, Reference Gericke and Eisenbart2017) in this Special Issue bridges this gap by combining multiple views on functional modeling.
Table 1. Recent decades of engineering function research
There are many representations available, such as the structure–behavior–function (Bhatta & Goel, Reference Bhatta and Goel1997), the function–behavior–structure (Qian & Gero, Reference Qian and Gero1996), the functional basis (Hirtz et al., Reference Hirtz, Stone, McAdams, Szykman and Wood2002), the function–behavior–state (Umeda et al., Reference Umeda, Ishii, Yoshioka, Shimomura and Tomiyama1996), affordance-based design (Maier, Srinivasan, et al., Reference Maier, Srinivasan, Vikram, Summers and Bernhard2007), the contact and channel model (Albers et al., Reference Albers, Thau and Alink2008), and the general function lists used in such design tools as morphological charts (Smith et al., Reference Smith, Richardson, Summers and Mocko2012), the house of quality (Olewnik & Lewis, Reference Olewnik and Lewis2005), and axiomatic design (Suh, Reference Suh1999). Developed by many researchers worldwide, each of these function representations and modeling approaches is characterized with a different intent, history, and context behind the representation (Erden et al., Reference Erden, Komoto, van Beek, D'Amelio, Echavarria and Tomiyama2008). It is because of the many different roles and uses within engineering design that these models have evolved disparately. Unfortunately, many of these approaches are limited by the “inventors” problem within design research, in which researchers will push the creation of a solution to a problem and a design need without actually designing the tool or method based on the intrinsic properties. Thus, many of the representations, while serving different specific purposes such as machinery and manufacturing systems with an emphasis on flows (Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013), might support other activities addressed by competing representations. This discrepancy suggests the need for developing a systematic comparison system.
Most of the modeling approaches were developed in response to a problem that would have been difficult to approach using methods with which the authors were familiar, and that work particularly well on certain problems. For example, although the functional models based on flow of energy, matter, and information work extremely well for production machinery (Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013), in which many examples are used to inform the model, they are ineffective in elucidating the functions a single complex component (e.g., the turbine blade of a jet engine). Further, although most of the functional modeling approaches in the literature use examples to illustrate their points, they neither discuss the scope of their approach nor reflect about the applicability to other classes of problems.
4. LEVELS OF COMPARISON
A thorough analysis of this problem must be predicated on a short discussion of the different levels at which function modeling, representation, and reasoning can be compared. Such comparisons may range from the fundamental core idea through to the use of models (Fig. 1). The most theoretical level of comparisons involves studying differences of “purpose,” “transformation,” or “intent” (Rosenman & Gero, Reference Rosenman and Gero1998), which have been reduced to formal frameworks such as the function–behavior–structure (Gero & Kannengiesser, Reference Gero and Kannengiesser2004). Such frameworks have been used to create defined representations for instantiating models, which are then coupled with reasoning activities, such as “model building” and “model using” for drawing inferences (Cebrian-Tarrason et al., Reference Cebrian-Tarrason, Lopez-Montero and Vidal2008). Tools such as FunctionCAD (Nagel et al., Reference Nagel, Perry, Stone and McAdams2009), 2nd-CAD (Vargas-Hernandez & Shah, Reference Vargas-Hernandez and Shah2004), function-behavior-state modeler (Umeda et al., Reference Umeda, Ishii, Yoshioka, Shimomura and Tomiyama1996), design repository (Bohm et al., Reference Bohm, Stone and Szykman2005), and ConMod (Sen et al., Reference Sen, Summers and Mocko2013b ) have been used to support such model-building activities. The models created from these tools have then been used for different inferences, such as failure modes (Stone et al., Reference Stone, Tumer and Van Wie2005), predicting assembly time (Mohinder et al., Reference Mohinder, Gill and Summers2017), and predicting assembly time (Mathieson et al., Reference Mathieson, Arlitt, Summers, Stone, Shanthakumar and Sen2011; Gill & Summers, Reference Gill and Summers2016). Thus, we must consider the primary factors of representation and reasoning, and their interaction with the modeler and user.
Fig. 1. Level of comparison and consideration during function benchmarking.
Different functional modeling approaches have different core ideas at the core of the approach. For example, some analyze function is terms of the flow (Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013), whereas other approaches involve elucidating the relationship of function, structure, and behavior (Gero & Kannengiesser, Reference Gero and Kannengiesser2004). Underpinning the core idea of any of these approaches are various notions of functions (Vermaas, Reference Vermaas2013).
The different modeling approaches usually are supplied with modeling formalisms, which are the parameters in which a functional model is described. For example, both verb–noun pairs (Hirtz, Stone, & McAdams, Reference Hirtz, Stone, McAdams, Szykman and Wood2002; Nagel, Stone, et al., Reference Nagel, Stone, Hutcheson, McAdams and Donndelinger2008; Nagel et al., Reference Nagel, Perry, Stone and McAdams2009) and either sentences or single words have been used to describe a given function (Deng, Reference Deng2002). Conversely, contact and channel have been used to prescribe the elements contained within the description of a function. One such formalism involves the description of at least two working surface pairs where the function is enacted (Albers et al., Reference Albers, Thau and Alink2008).
The expressive power of these functional models is greatly affected by such formalisms that are used in their construction, a variation of great importance when building functional models of complex products, where the effort increases significantly with the complexity of a model. Some modeling approaches are characterized by a hierarchical decomposition of functions while others are not (Erden et al., Reference Erden, Komoto, van Beek, D'Amelio, Echavarria and Tomiyama2008). The models also vary in the degree of abstraction required; for example, contact and channel and function modeling requires a concrete embodiment of the modeling link functions, whereas functional descriptions are abstract in other models. This variation in turn affects the hierarchical decomposition as the lower level description might depend on the chosen embodiment.
The other dimension of comparison is the reasoning with functions both in building models and in using models once they have been generated, which is of importance when the models are used by someone other than their author. In such a case, the familiarity of the modeling approach and the intuitiveness of the representation then becomes a major issue when inferring information. Several studies have been undertaken to study the authorship and consistency and interpretability of models (Kurfman et al., Reference Kurfman, Stock, Stone, Rajan and Wood2003; Caldwell, Ramachandran, et al., Reference Caldwell, Ramachandran and Mocko2012; Caldwell, Thomas, et al., Reference Caldwell, Thomas, Sen, Mocko and Summers2012), as well as elucidating the correctness of model construction (Nagel et al., Reference Nagel, Bohm, Linsey and Riggs2015). Alternatively, some approaches have been proposed that automatically reason on function models from database collections (Lucero et al., Reference Lucero, Viswanathan, Linsey and Turner2014; Patel, Andrews, et al., Reference Patel, Andrews and Summers2016; Sridhar et al., Reference Sridhar, Fazelpour, Gill and Summers2016). Finally, some approaches entail the support of first principle based physics reasoning (Goel et al., Reference Goel, Rugaber and Vattam2009; Sen et al., Reference Sen, Summers and Mocko2011b , Reference Sen, Summers and Mocko2013a ). Thus, it is possible that reasoning might be supported through human use and interpretation or through automated reasoning to infer information.
Some functional modeling approaches, such as functional basis (Hirtz et al., Reference Hirtz, Stone, McAdams, Szykman and Wood2002), are supported by dedicated modeling tools (Vargas-Hernandez & Shah, Reference Vargas-Hernandez and Shah2004; Bryant et al., Reference Bryant, McAdams, Stone, Kurtoglu and Campbell2006; Nagel et al., Reference Nagel, Perry, Stone and McAdams2009), which allow the user to build models more comfortably and therefore also support building larger models. The usability of the tools affects the potential success of the modeling approach independent of the other elements.
5. COMPARISON ACROSS THE CRITERIA
The framework in Figure 1 proposes different criteria on which modeling approaches can be compared. Here we detail our criteria for comparing these criteria by proposing characteristics along which the comparison can be made.
5.1. Representation characteristics
When comparing function modeling approaches, the typical approach initially involves comparing the representations at the formalism level, including vocabulary and grammar. A representation is the formalism through which a model is constructed, which means that a model is instantiated through a representation. Multiple models of the same “real-world” target may be created through the same representation with each distinct model clearly mapped to a single real-world target. Ambiguity is introduced when one model serves as a surrogate for multiple targets (Shah & Mantyla, Reference Shah and Mantyla1995), which are not clearly specified.
Similar comparison criteria, derived from research in artificial intelligence, include representational adequacy, inferential adequacy, inferential efficiency, and acquisitional efficiency (Winston, Reference Winston2005). Another approach to compare representations examines the vocabulary, structure, expression, purpose, and abstraction (Summers & Shah, Reference Summers and Shah2004). Expanding upon that research, we propose that the representation comparison should include, but not be limited to the following:
-
• scope: the domain for which the function modeling approach is intended (Nagel, Vucovich, et al., Reference Nagel, Vucovich, Stone and McAdams2008);
-
• flexibility: the ability to modify and adapt the representation to address new problems (Regli et al., Reference Regli, Hu, Atwood and Sun2000);
-
• indexing: support access to the right (or useful) knowledge when needed (Goel & Bhatta, Reference Goel and Bhatta2004);
-
• consistency: enforce physics and other consistencies (Sen et al., Reference Sen, Summers and Mocko2011b );
-
• translationabilty: tied to other engineering models (Nebel, Reference Nebel2000);
-
• behavior: ability of the representation to simulation behavior (Qian & Gero, Reference Qian and Gero1996); and
-
• scalability: support both simple and complex problem types (Chiang et al., Reference Chiang, Pennathur and Mital2001).
5.2. Modeling characteristics
In addition to these representational issues, the interaction of the designer during the model construction is of concern when comparing the functional modeling approaches. For instance, is the modeling computationally supported, restricted to human effort, or is a mixed initiative approach supported (Sen et al., Reference Sen, Summers and Mocko2013a ). In addition, the support of various construction approaches within the model, such as forward chaining (moving from input to output), backward chaining (moving from output to input), nucleation, environment to system (outside to inside), or system to environment (inside to outside) are also important considerations (Sen & Summers, Reference Sen and Summers2014). A final characteristic relates to the support of decomposition and recomposition across multiple hierarchical levels and abstractions within the respective modeling approach (Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013).
5.3. Cognitive criteria characteristics
The concept of cognitive criteria has been developed in human–computer interaction to help software designers to think through the usability of the artifacts they were creating, such as programming languages or user interfaces (Green, Reference Green, Diaper and Hammond1991). Although many developers and software engineers have experience in developing well-design information artifacts, they have no way of articulating why these approaches are appropriate for meeting user needs. Functional modeling approaches are considered information artifacts, similar to how programming languages and cognitive criteria offer a vocabulary for discussing usability issues, which is informed by cognitive science (Blackwell et al., Reference Blackwell, Britton, Cox, Green, Gurr, Kadoda, Kutar, Loomes, Nehaniv, Petre, Beynon, Nehaniv and Dautenhahn2001). The framework is deliberately broad to avoid being overwhelmed in the details of an implementation and thereby losing the sought-after conceptual improvements. However, this task-specific approach, which addresses processes and activities rather than merely the final product, means that it can be used to evaluate and not simply compare functional modeling approaches. The cognitive criteria are orthogonal in supporting reasoning trade-offs and for analyzing the space of possible solutions in a coherent manner, and where possible observing the effect of combinations of criteria. Table 2 presents a selection of the cognitive criteria with their questions for programming and a possible interpretation of these questions for functional modeling, which would require refinement prior to a benchmarking exercise (Green & Petre, Reference Green and Petre1996).
Table 2. Key cognitive dimensions (criteria) based on Green and Petre (Reference Green and Petre1996)
The computing cognitive criteria have a dimension of the progressive evaluation, which is the method for obtaining feedback on the modeling through the process and that appears far more meaningful for a programming language, which can be deployed in many different ways. Criteria such as diffuseness, which address the number of symbols or graphic entities required to express a meaning, and hard mental operations, which questions the need for annotations, also specifically address the notion. Both greatly depend upon a particular implementation version, as few standards concerning functional modeling have yet to emerge.
For each of the dimensions a scale of subcategories can be developed such as the abstraction gradient, which is decomposed into abstraction-hating, abstraction-tolerant, and abstraction-hungry (Green & Petre, Reference Green and Petre1996). For example, while abstraction-hungry programming languages may be considered difficult, abstraction can reduce error proneness and increase viscosity.
5.4. Reasoning characteristics
Reasoning is the comparison dimension that motivates the need for a common, standard benchmark for evaluating function modeling approaches. It is for different classes of reasoning that each function model is constructed. These reasoning activities entail failure detection (Kurtoglu & Tumer, Reference Kurtoglu and Tumer2008), reverse engineering and product understanding (Hirtz et al., Reference Hirtz, Stone, McAdams, Szykman and Wood2002), design decision justification (Gero, Reference Gero1996), design verification and validation (Wiltgen & Goel, Reference Wiltgen and Goel2016), or concept definition and exploration (Pahl et al., Reference Pahl, Beitz, Blessing, Feldhusen, Grote and Wallace2013). Some types of reasoning that can be evaluated with respect to support include the following:
-
• Interpretability: How consistent and precise is the interpretation of the function models across different individuals, domain, and expertise (Caldwell, Thomas, et al., Reference Caldwell, Ramachandran and Mocko2012)?
-
• Physics maintenance: Can questions about conservation of energy or material, irreversibility, or other physics queries be answered (Sen et al., Reference Sen, Summers and Mocko2011a )?
-
• Analogical mapping: Does the representation support analogical mapping and alignment (Qian & Gero, Reference Qian and Gero1996)?
-
• Pattern learning: Does the representation support the learning of abstractions required for analogical transfer (Bhatta & Goel, Reference Bhatta and Goel1997)?
-
• State transformations: Does the representation support answering questions about different states (Deng, Reference Deng2002)?
-
• Change propagation: Does the representation support discovery about the effects of perturbations in the system (Kurtoglu & Tumer, Reference Kurtoglu and Tumer2008)?
These reasoning criteria might relate to the cognitive criteria. For example, interpretability, analogical mapping, and change propagation might relate to closeness of mapping and viscosity. In contrast, physics maintenance and state transformation is focused more on the content of the model. Other challenges in reasoning might relate to the ability to contextualize the system within a larger environment or the distribution of system-level functions to several distributed elements.
6. BENCHMARK PROBLEM TYPES
In order to explore the different characteristics of representation that enable cognitive criteria to support reasoning activities, a set of benchmark problems are needed. Expanding upon the problem type classifications in the literature, we propose four types for study, an example for each type found in the literature, and a list of alternative examples for each. We do not include large-scale, complex systems such as submarines, aircraft, or space systems as these are not readily available to all researchers for benchmarking activities. A limitation of many benchmarking efforts is that the problems selected should be relatively simple so that researchers are not dissuaded from applying their approaches against similar problems. Complex systems might be of interest in benchmarking, but the effort involved in constructing these models might be too great when compared to the value of understanding the differences. This challenge is noted here but is not resolved.
6.1. Reverse engineered products
Many function modeling approaches have been demonstrated on existing products after dissection and reverse engineering. A repository of commercial products that have been reversed engineered to understand them has been developed with the function representation serving as the foundation for the information model (Bohm et al., Reference Bohm, Stone and Szykman2005). An advantage of including this type of problem in the benchmarking formalism is that the products exist, and their performance can be measured and evaluated. A reverse engineered product provides a common platform for comparison. An example product that has been used extensively (Maier, Ezhilan, et al., Reference Maier, Ezhilan, Fadel, Summers and Mocko2007; Huang & Jin, Reference Huang and Jin2009; Hamraz et al., Reference Hamraz, Caldwell and Clarkson2012) to explore function modeling is the hairdryer (Figs. 2 and 3). Other possible products that can be considered are pneumatic impact drivers (a greater number of mechanical components), battery power tools (readily available in multiple variations), vacuum cleaners (for comparison across multiple customer cultural differences), bike lights (simple and inexpensive systems), or the glue gun (simple product with material flow) proposed for this Special Issue (Mocko et al., Reference Mocko, Summers, Teegavarapu, Ezhilan, Maier and Fadel2007; Summers et al., Reference Summers, Eckert and Goel2017). The selection of a common product is most important so that the community can standardize their demonstration cases.
Fig. 2. Example knowledge types in the hairdryer (Hamraz et al., Reference Hamraz, Caldwell and Clarkson2012).
Fig. 3. Function structure of a hairdryer product stored in the Design Repository (http://repository.designengineeringlab.org/).
6.2. Familiar product
Engineers use reverse engineering and the dissection of products to map existing systems and components to specific functionality, the first step of which involves hypothesizing the internal functioning of a product (Otto & Wood, Reference Otto and Wood1998). Further, while reverse engineering can test the ability of a representation to model the detailed functionality of an existing system, modeling a familiar without the product in hand can expose the ability to be fluidly and flexibly model the system, as significant backtracking and hierarchical jumping is likely. An example of the results from an experimental exercise to explore how engineers model known products is found in Figure 4 (Eckert et al., Reference Eckert, Alink, Ruckpaul and Albers2011). Different engineers are likely to model the system in different ways even given the same underlying representation, so the expressive power of modeling approaches can be assessed. Thus, this benchmark product can be used to explore defining characteristics of the representation, such as consistency and repeatability, without the cost of buying products to reverse engineer. Other products considered for reverse engineering are bicycles, gear boxes, or printing machines.
Fig. 4. Example function model for a hydraulic pump (Eckert et al., Reference Eckert, Alink, Ruckpaul and Albers2011).
6.3. Novel products
Generative forward system design involves the development of new multicomponent systems for characteristics not previously addressed: novelty (not yet attempted), system (multiple components), and intentional (design with a purpose). Examples of “problems” that used to benchmark and compare different function modeling approaches include automated omelet makers, hand-cranked pretzel makers, shoe-string tying mechanisms, clothes folding machines in hotel laundries, and a hand-cranked automated burrito maker. Benchmark examples can be drawn from literature to support the objectivity of the benchmark. For example, the burrito folding system problem has been used previously in the comparing of function lists and function structures in morphological charts (Richardson et al., Reference Richardson, Summers and Mocko2011). Figure 5 illustrates the function lists and structures for the burrito folder that were used in these ideation experiments. Such a benchmark is useful in exploring the degree to which a function representation can be used in understanding novel problems and generating new solutions.
Fig. 5. Function list and function structure for burrito folder (Richardson et al., Reference Richardson, Summers and Mocko2011).
6.4. Single-component products
Single-component multifunctional products, such as passive morphing airfoils (Schultz et al., Reference Schultz, Sen, Caldwell, Mathieson, Summers and Mocko2010) and speed screws are also useful as benchmarks (Albers et al., Reference Albers, Thau and Alink2008). For example, the design and analysis of a speed screw demonstrated the effective use of the contact and channel model in the design of a single component. Given that function models cannot capture the functionality and behaviors associated with single components, however, this scaling ability in both small and large systems must be explored. While the speed screw benchmark example (Fig. 6) shows the downward scalability of reverse engineering, the passive, morphing airfoil design (Fig. 7) illustrates the downward scalability of forward engineered products. Larger scale systems, such as aircraft, are not considered within the benchmarking protocol due to the challenge of general access for the researchers.
Fig. 6. States of working surface pairs for the contact and channel model for a speed screw (Albers et al., Reference Albers, Thau and Alink2008).
Fig. 7. Function structure for a morphing airfoil (Schultz et al., Reference Schultz, Sen, Caldwell, Mathieson, Summers and Mocko2010).
6.5. Problem characteristics
While these problem types are focused on the “thing” to be modeled within the function representations, other characteristics may also be considered when comparing the problems. This list is not exhaustive and is intended to recommend to the community other criteria to be considered when developing the benchmark problems for experimental studies. Defining these characteristics will enable the development of similar problems to capture and address comparable criteria.
For instance, measures of the problem size, problem connectivity, and problem difficulty are used to compared problem complexity (Summers & Shah, Reference Summers and Shah2010). Similarly, an information metric was used to compare size complexity (i.e., different problems and prototypes) to explore the impact of representation and level of fidelity in assessing requirement satisfaction (Hannah et al., Reference Hannah, Joshi and Summers2011). Finally, problem size as measured with a requirement count and number of words was used to compare design problems in a function modeling experiment (Worinkeng et al., Reference Worinkeng, Joshi and Summers2015; Patel, Kramer, et al., Reference Patel, Kramer, Summers and Shuffler-Porter2016).
Other considerations used in defining a design problem included the following:
-
• the domain appropriateness for the participants (Ostergaard et al., Reference Ostergaard, Wetmore, Divekar, Vitali and Summers2005; Wetmore et al., Reference Wetmore, Summers and Greenstein2010; Sen & Summers, Reference Sen and Summers2014);
-
• realistic and not contrived concepts (Linsey et al., Reference Linsey, Tseng, Fu, Cagan, Wood and Schunn2010; Thiagarajan et al., Reference Thiagarajan, Patel, O'Shields and Summers2017);
-
• constraints and conditions of the problem (Stacey & Eckert, Reference Stacey and Eckert2010; Eckert & Stacey, Reference Eckert and Stacey2014);
-
• a compelling and intrinsic motivation for the participants (Linsey et al., Reference Linsey, Tseng, Fu, Cagan, Wood and Schunn2010; Joshi & Summers, Reference Joshi and Summers2014); and
-
• representation characteristics of both expression and abstraction (McKoy et al., Reference McKoy, Vargas-Hernández, Summers and Shah2001; Summers & Shah, Reference Summers and Shah2004; Hannah et al., Reference Hannah, Joshi and Summers2011).
The authors wish to emphasize that the problems developed should be common across multiple criteria for the cross-comparison between benchmarking results, particularly when using sampling logic for the experimental analysis.
7. RESEARCH RECOMMENDATIONS
While this paper is written as a call to action for the establishment of standards for function modeling benchmarking, such standards should be developed and evolved with input from both users and developers of the various modeling approaches. Such collaborations should also occur in an academically open environment. Researchers will find clear value in this benchmarking process as it forces the disparate communities to begin to communicate with each other, distributes tutorials on the creation and execution of a variety of models and methods to enhance the education of future engineers, and can be paired with a reasoning/representation selection database to systematically develop informed tools and methods. The benchmarking exercises can also help researchers justify a systematic evolution of their approaches.
We do not see benchmarking as an alternative to validation and verification of functional modeling approaches, but as an addition, which can help in the process of evaluating the approach. While validation and verification can attest to the completeness and correctness of a functional modeling approach, benchmarking helps us compare multiple approaches in terms of information captured, reasoning supported, computational efficiency, and cognitive plausibility.
An internationally diverse benchmarking development group representing the various approaches is a recommended strategy for creating this protocol. In addition to the intellectual motivation for benchmarking, this group can pressure both public and private sector agencies to allocate research funds to support the development of benchmarking schemes.
These benchmarking criteria described above are not intended to serve as the final set, but rather to serve as the point of departure. Additional study is needed to refine these characteristics and to strategically select case examples for comparison. This proposed development group can support these endeavors directly.
Joshua D. Summers is a Professor of mechanical engineering at Clemson University, where he is also the Co-Director of the Clemson Engineering Design Applications and Research Group. He earned his PhD in mechanical engineering from Arizona State University and his MS and BS the University of Missouri. Dr. Summers worked at the Naval Research Laboratory (VR Lab and NCARAI). His research has been funded by government, large industry, and small- to medium-sized enterprises. Joshua's areas of interest include collaborative design, knowledge management, and design enabler development with the overall objective of improving design through collaboration and computation.
Claudia Eckert is a Professor of design at Open University. She has a longstanding interest in studying and supporting industrial practice in different design domains and has published numerous papers on it. In particular, she has been working on process modeling, engineering change, and functional modeling of complex engineering products.
Ashok K. Goel is a Professor of computer science in the School of Interactive Computing at Georgia Institute of Technology. He is the Director of the School's PhD program in Human-Centered Computing and the Design & Intelligence Laboratory. Ashok is also Co-Director of the Institute's Center for Biologically Inspired Design and a Fellow of the Brooke Byer's Institute for Sustainable Systems. He served on the board of directors of the Biomimicry Institute from 2012 to 2017 and was its President from 2015 to 2017. Dr. Goel has been conducting research into artificial intelligence, cognitive science, and human-centered computing for 30 years, with a focus on computational design, modeling, and creativity. His 2012 TEDx@Peachtree talk summarized some of his research on computational design and creativity. An interactive tool for supporting some aspects of biologically inspired design was developed by his laboratory. Ashok is Editor of AAAI's AI Magazine and an Associate Editor of the Design Research Society's Design Science Journal as well as Artificial Intelligence for Engineering Design, Analysis and Manufacturing.
