2.1. Idea generation approaches

Recommended practices in idea generation encourage that multiple, diverse concepts are generated and considered (Zenios et al. Reference Zenios, Makower, Yock, Brinton, Kumar, Denend and Krummel2009). By creating a large number of diverse ideas, engineers are more likely to generate nonobvious solutions (Zenios et al. Reference Zenios, Makower, Yock, Brinton, Kumar, Denend and Krummel2009). Additionally, diverse ideas support broader perspectives on solutions and support deeper consideration of the real problem (Dorst & Cross Reference Dorst and Cross2001). Diverse ideas can include unconventional ideas, and these ideas can stimulate novel solutions that have not been previously considered (Kelley & Littman Reference Kelley and Littman2001). Idea generation recommended practices also encourage limiting early evaluation and documenting any new idea even if it seems impractical, as a “wild” idea could be transformed into a successful solution and inspire other ideas (Kelley & Littman Reference Kelley and Littman2001).

Both novice and experienced engineers have been shown to struggle to implement recommended design practices. Novice engineers have difficulty generating and considering multiple ideas; novices often limit the diversity of ideas by focusing on a particular concept or variations of similar ideas, a term called fixation (Purcell & Gero Reference Purcell and Gero1996). In addition to fixating on a particular idea type, novices can fixate on early ideas even when they realize that these ideas have major flaws (Ball, Evans & Dennis Reference Ball, Evans and Dennis1994). Even expert engineers have been shown to struggle to break away from existing, well-known solutions (Linsey et al. Reference Linsey, Tseng, Fu, Cagan, Wood and Schunn2010) and evaluate ideas too early (Kelley & Littman Reference Kelley and Littman2001).

To support engineering designers in achieving recommended practices in idea generation, the use of ideation structures and tools is recommended. For example, brainstorming “rules” provide a structure for how groups should collect ideas, by building off other suggestions and not limiting the types of ideas (Osborn Reference Osborn1963). Ideation tools have been shown to promote quantity, creativity, diversity and elaboration of ideas generated (Linsey, Wood & Markman Reference Linsey, Wood and Markman2008; Hernandez et al. Reference Hernandez, Schmidt and Okudan2013; Daly et al. Reference Daly, Seifert, Yilmaz and Gonzalez2016; Lee et al. Reference Lee, Daly, Huang-Saad, Seifert and Lutz 2018a,Reference Lee, Ostrowski, Daly, Huang-Saad and Seifert b). Examples of tools include Brainwriting (Heslin Reference Heslin2009), Design Heuristics (Daly et al. Reference Daly, Yilmaz, Christian, Seifert and Gonzalez2012b), IDEO cards (IDEO 2002), Morphological Analysis (Allen Reference Allen1962), TRIZ (Altshuller Reference Altshuller1997) and Wordtree Design-by-Analogy (Linsey et al. Reference Linsey, Wood and Markman2008). Some tools may be better suited for achieving particular goals (i.e., some tools may best limit fixation while others improve the number of ideas generated). Structures and tools are sometimes specifically meant for group ideation (i.e., Brainwriting) while others support individual ideation. Group idea generation can benefit ideation, but individual ideation is recommended before group ideation (Diehl & Stroebe Reference Diehl and Stroebe1987). While studies have examined the impacts of specific structures and tools, few studies have focused on the extent to which students aim to employ recommended practices in their approaches and which tools are used during idea generation.

2.2. Idea development approaches

Engineers employing recommended practices in idea development iterate on early ideas to improve their potential. This iteration includes elaborating on existing ideas, building new ideas inspired by existing ones and generating new types of ideas based on gaps identified within existing ideas (McMahon et al. Reference McMahon, Ruggeri, Kämmer and Katsikopoulos2016). Engineers often iterate to modify ideas to address inconsistencies or errors, improve solutions to optimize certain characteristics and integrate multiple ideas to develop new ideas (Adams & Atman Reference Adams and Atman1999). Furthermore, engineers may ask for feedback from their stakeholders to inform where ideas need further iteration (Sanders & Stappers Reference Sanders and Stappers2008).

In practice, novice engineers have been shown to limit idea development and focus on evaluating and selecting an idea for pursuit (Crismond & Adams Reference Crismond and Adams2012). If they do engage in some development, they focus on developing a single idea by refining the same solution and adjusting the details of that solution and thus do not consider other options (Cross Reference Cross2008). Novices engage in minimal iteration of ideas as compared to experts (Atman et al. Reference Atman, Chimka, Bursic and Nachtmann1999) and solve design problems as a linear process that can be done only once (Crismond & Adams Reference Crismond and Adams2012), leaving very little room to explore beyond their initial ideas.

There are few strategies and tools for idea development discussed in design literature, as design methods emphasize idea generation and selection (Dubberly Reference Dubberly2004; Cross Reference Cross2008). One existing strategy that has been demonstrated to support idea development is Brainstorming in small groups (McMahon et al. Reference McMahon, Ruggeri, Kämmer and Katsikopoulos2016), which encourages building on initial ideas without early evaluation. Other group members can use the initial ideas to develop more complete ideas and combine features of multiple ideas. Additionally, some idea generation tools have been explored as idea development tools. For example, Design Heuristics was shown to support students in elaborating or further specifying their design ideas (Christian et al. Reference Christian, Daly, Yilmaz, Seifert and Gonzalez2012; Kramer et al. Reference Kramer, Daly, Yilmaz, Seifert and Gonzalez2015; Clancy et al. Reference Clancy, Murphy, Daly and Seifert2023). Also, Design Heuristics helped students to consider additional features and transform their previous ideas to further develop their ideas. C-Sketch in a group setting has been shown to support idea development as group members add modifications to previous ideas produced by other group members (Shah et al. Reference Shah, Vargas‐Hernandez, Summers and Kulkarni2001). Although research has demonstrated the usefulness of specific tools in supporting idea development, students’ natural idea development practices have been underexplored in the research literature.

2.3. Idea selection approaches

During idea selection, engineers evaluate numerous ideas and select promising ideas (Kudrowitz & Wallace Reference Kudrowitz and Wallace2013). Recommended practices encourage designers to appropriately evaluate and select ideas by balancing benefits and trade-offs by articulating both the positive features as well as drawbacks (Crismond & Adams Reference Crismond and Adams2012). Recommended practices also suggest ideas to be selected after employing back-of-the-envelope estimated calculations to ensure that their concepts meet functional requirements (Brand Reference Brand1995).

While various idea generation and development tools can help in exploring a design solution space, innovative ideas are often filtered out during idea selection (Rietzschel et al. Reference Rietzschel, Nijstad and Stroebe2006). Both novice and expert designers who select poor concepts have large costs associated with redesign, while designers who select high-quality concepts increase their likelihood of product success (Huang et al. Reference Huang, Liu, Li, Xue and Wang2013). Expert engineers often select concepts that are conventional or have shown success in the past instead of novel ones (Ford & Gioia Reference Ford and Gioia2000). Also, Toh & Miller (Reference Toh and Miller2015) found that novice engineers focused on technical feasibility and effectiveness at the cost of originality. Inherent bias against unconventional ideas exists due to the risk and uncertainties of unconventional ideas (Rubenson & Runco Reference Rubenson and Runco1995). Although innovation is emphasized in idea generation, both novice and expert engineers often filter out innovative ideas during concept selection to minimize risk.

To support designers in idea selection, various formalized methods have been developed, including the Analytical Hierarchy Process (Marsh Reference Marsh1993), Pugh’s evaluation method (Pugh Reference Pugh1991) and Utility Theory (Pahl & Beitz Reference Pahl and Beitz1991). These methods assign attribute values to compare the characteristics of design options to find an optimal solution. Studies have examined the usefulness and effectiveness of idea selection tools (Starkey, Gosnell & Miller Reference Starkey, Gosnell and Miller2015; Zheng & Miller Reference Zheng and Miller2016; Zheng, Ritter & Miller Reference Zheng, Ritter and Miller2018). Studies compared how the Tool for Assessing Semantic Creativity (TASC) and Concept Selection Matrix (CSM) influenced students’ decision-making process during idea selection and identified that students are more likely to select ideas that are ranked highly with the CSM method (Zheng & Miller Reference Zheng and Miller2016; Zheng et al. Reference Zheng, Ritter and Miller2018). Another study examined the effects of TASC and the Shah Vargas-Hernandez and Smith (SVS) method and concluded that TASC may be a means to remove biases as the group evaluated the creativity of ideas (Starkey et al. Reference Starkey, Gosnell and Miller2015). However, few studies have examined engineering students’ natural idea selection practices and their thought processes.

2.4. Development of hybrid learning blocks to support idea generation, development and selection

Teaching idea generation, development and selection that supports innovation is challenging for educators (Grasso et al. Reference Grasso, Burkins, Helble and Martinelli2010). Often, developing skills in generating and pursuing innovative concepts are left to students to figure out, rather than implementing specific techniques taught in class (Dym et al. Reference Dym, Agogino, Eris, Frey and Leifer2005). Also, instructors who desire to provide more explicit instructions have indicated challenges because of the amount of effort involved in providing the education their students require (Richards & Carlson-Skalak Reference Richards and Carlson-Skalak1997). The curriculum is slow to change and restructuring an existing course or adding a new course to focus on specific design skills can be a long process that requires significant investment from faculty members.

To provide support for some of these challenges in design education, the Center for Socially Engaged Design developed Hybrid Learning Blocks that provide content, assessment and practice with specific design skills. The Center for Socially Engaged Design defines its design approach as a contextually rich integration of human, cultural, economic and environmental factors within the processes of designing technology. The Center has a significant focus on educating students in comprehensive skillsets required for this type of engineering problem-solving, including creativity, stakeholder engagement and contextual integration. The Hybrid Learning Blocks were an education resource developed by the Center in response to educational challenges engineering students faced in tending to the social and creative aspects of engineering work (Young et al. Reference Young, Daly, Hoffman, Sienko and Gilleran2017). They were developed to be used by cocurricular student organizations as well as by instructors within their courses. Hybrid Learning Block topics include, for example, conducting design interviews, performing observations, eliciting user requirements, developing engineering specifications, generating concepts, developing concepts and selecting concepts.

The Hybrid Learning Blocks integrate asynchronous online learning with personalized coaching and feedback. The asynchronous learning blocks were built to facilitate access to learning materials without time and place constraints. Since students working on design projects can be at different stages of their work, providing asynchronous learning modules allows them to access specific modules as needed. They are also easily adapted to synchronous versions, however, their development as asynchronous resources offers flexibility to users. Each learning block leverages the same overall structure, consisting of five key elements (as shown in Figure 1): (1) Prior Knowledge Review gauges students’ preconceptions and existing knowledge on the topic; (2) Core Content provides recommended practices on a particular design topic using readings and videos; (3) Knowledge Check uses a combination of closed- and open-ended problems to evaluate students’ learning; (4) Application prompts students to apply the concepts from the learning block to a real-life scenario and receive feedback from a coach to evaluate their skills and (5) Block Reflection allows students to reflect on their learning experience and challenges students on their pre-existing ideas about the topic.

Figure 1. The hybrid learning block model.

3.1. Recruitment

Twenty-one undergraduate mechanical engineering students were recruited for the study, which resulted in over 30 hours of think-aloud and interview data. This number of student participants is appropriate for an in-depth qualitative study (Creswell Reference Creswell2013) and is similar to other qualitative design studies (Kim et al. Reference Kim, Chung, Beckman and Agogino2016; Lauff, Kotys-Schwartz & Rentschler Reference Lauff, Kotys-Schwartz and Rentschler2018). Student participants were recruited on a rolling basis and the researchers documented consistent behaviors with 21 students, indicating saturation – defined as no additional themes emerging as additional participant data are added and further data collection may not be necessary (Creswell Reference Creswell2013) – had been achieved.

The student participants were recruited through targeted emails to undergraduate mechanical engineering students at a large Midwestern university. All student participants had taken at least one design-related college course where they gained experience in idea generation, development and selection. Also, many student participants had design-related internships or cocurricular design activities. Thus, they had multiple exposures to design and had the opportunity to develop strategies to employ in idea generation, development and selection. Student participants’ background information with anonymized names is included in Table 1.

Table 1. Participant demographics

3.2. Data collection

Student participants completed a design task without being provided any suggested strategies to leverage and were interviewed afterward. The design task asked students to develop solutions to a given problem statement and select a final solution at the end. Student participants completed this task using whatever approaches they wanted, in order to investigate their natural tendencies in a nonguided setting. They were asked to spend a minimum of an hour working on the design task using any resources they needed. Student participants often requested to use a computer or phone with the internet.

Student participants were asked to think aloud throughout the session as they wrote and completed the design task. The think-aloud data were recorded using a Livescribe Echo pen. The think-aloud protocol asks participants to verbalize their thought processes during a problem-solving task (van Someren, Barnard & Sandberg Reference Van Someren, Barnard and Sandberg1994). Think-aloud approaches capture processes and ideas in a person’s working memory rather than their long-term memory (Ericsson & Simon Reference Ericsson and Simon1980). Working memory provides a more accurate representation of their processes as they engage in a task as compared to recalling information after completing a problem-solving event.

The problems for the design task were developed based on several criteria. Solutions to these problems needed to be product-oriented since we planned the experiment with mechanical engineering students. The problems were developed to minimize the expertise needed in a particular context to ensure that students did not need extensive knowledge to generate ideas. We modified three existing tasks used in other studies that had similar criteria (Rechkemmer et al. Reference Rechkemmer, Makhlouf, Wenger, Silk, Daly, McKilligan and Jablokow2017) and we conducted two rounds of pilot tests to refine the language. After the pilot tests, the two tasks we selected were: (1) the low-skill snow transporter problem that asks students to design a personal tool for people who lack ski and snowboard experience, and (2) the one-handed opener for lidded food containers problem that asks students to develop a way for people with limited use of one upper extremity to open a lidded food container. The full problem descriptions are included in the Supplementary Material.

After the design task, the student participants were interviewed using a semi-structured interview protocol. Interviews allow for the exploration of perceptions and opinions and enable probing for more information, which helps ensure the validity of the data because it allows for clarification of responses (Hutchinson & Wilson Reference Hutchinson and Wilson1992) and more complete information (Bailey Reference Bailey1994). The interview questions were developed through multiple iterations. Open-ended questions were constructed, following recommended practices in interview protocols (Jacob & Furgerson Reference Jacob and Furgerson2012), to understand students’ idea generation, development and selection practices, and questions were framed neutrally to avoid expressing personal opinions and leading interviewees, consistent with recommended practices (Patton Reference Patton2015). Examples of questions included: How did you generate ideas to address the problem? How, if at all, did you iterate on any of your ideas? Can you tell me about how you selected your final idea? Prior to using the protocol for data collection, one pilot interview was conducted to ensure clarity. Each interview was audio-recorded for analysis.

3.3. Data analysis

The think-aloud and interview data were transcribed for analysis, and student participants’ sketched data were matched with think-aloud data. The think-aloud data were analyzed to uncover student participants’ approaches by first considering the data with a list of deductive codes for each idea phase developed based on previously documented behaviors in idea generation, development and selection such as “focused on existing solutions to the problem” for the idea generation phase and “balancing benefits/trade-offs” for the selecting ideas phase (Crismond & Adams Reference Crismond and Adams2012). The interview data was considered additional information to support the think-aloud data as student participants often elaborated on their processes.

After labeling behaviors according to our pre-defined set, inductive codes were added based on recurring trends in the data to form the complete set of codes to describe student participants’ behaviors for each idea phase. For example, an inductive code of “assumed additional requirements” was added to the codebook as a recurring pattern to describe the approach of adding additional requirements not stated in the problem. Table 2 includes the list of codes to describe student behaviors during Phase 1. After the codebook was finalized, a third coder independently coded the interviews and think-aloud sessions and compared codes to one of the other coders. Inter-rater reliability for behavior codes was calculated as 75%. Values greater than 70% are typically acceptable for inter-rater reliability (Osborne Reference Osborne2008). The coders discussed all discrepancies and reached a full agreement prior to finalizing the findings.

Table 2. Frequency of student participants’ idea generation, development and selection behaviors during Phase 1

^a Added codes through inductive coding.

4.1. Phase 1 idea generation approaches

When student participants were given the freedom to approach ideation how they chose, student participants (1) assumed additional requirements that were not explicitly described in the problem statement that limited divergence in ideation, (2) immediately evaluated and/or eliminated ideas after generation, (3) looked for existing solutions to the problem, (4) did not utilize any idea generation strategies and (5) focused on the practicality of ideas as a goal.

When student participants were given a design problem, they used the stated constraints from the design problem as a guide and assumed additional requirements that were not part of the problem statement. For example, Henry was working on a one-handed opener problem and indicated an additional requirement that was not stated in the problem:

By creating an additional requirement, Henry only came up with ideas where the container was constrained to open with one hand.

Similarly, Isaac came up with an assumed requirement not included in the problem statement. He was tasked with the low-skill snow transport problem that prompted him to design a personal transportation method on snow, but he added:

By focusing on a device that can stand on the snow, he focused on devices that emphasize on balancing the user.

Student participants immediately evaluated and eliminated possible solutions during early concept generation. For example, Isaac was working on the snow transportation problem. He initially thought of an idea to use dogs to pull a sled to transport people on snow. However, he considered the weaknesses of the idea and immediately discarded it:

Victoria was working on the snow transportation problem and quickly discarded a solution that may be pricey and time-consuming to learn for the user, which led her to not consider an idea in the early stage:

By evaluating ideas immediately after generation, student participants did not document their ideas on paper and did not consider them as a possible solution.

Student participants also relied on current, existing solutions and minimally diverged, limiting their consideration of many alternatives. Since student participants were allowed to use any resource they needed, they searched on Google for existing solutions to the problem, which led to conventional solutions. For example, Andrea who was working on the one-handed container opener said:

Similarly, Steve focused on solutions he identified from an online search:

None of the student participants used any specific ideation strategies to support them in creating diverse alternatives. In the interviews, several student participants indicated that they were aware of ideation techniques but did not apply them to support ideation. For example, in an interview, Brian said:

Similarly, another student participant mentioned the knowledge of idea generation tools learned from previous exposure to design but did not implement them during the task:

Student participants had access to the internet and they could have leveraged online resources; student participants knew of idea generation techniques but did not apply them.

Student participants also emphasized coming up with feasible and existing solutions that meet all their requirements as a goal for idea generation. For example:

Student participants focused on coming up with existing and feasible solutions at the sacrifice of diversity, which can lead to conventional ideas. Recommended practices in idea generation encourage designers to come up with novel, unconventional ideas that can be used to inspire new ideas. Focusing on the feasibility of ideas early in concept generation can reduce the diversity of concepts considered.

4.2. Phase 1 idea development approach

Participants showed minimal signs of developing ideas further than their initial generation. For example, student participants indicated that he did not expand on his initial ideas to make improvements:

Student participants indicated that after they generated their ideas, they moved directly to finalizing their solutions.

4.3. Phase 1 idea selection approaches

Student participants frequently did not have to engage in idea selection as they focused on one idea throughout their design session. For example, Olivia, who was working on the one-handed opener problem, indicated that she had one idea for her solution and focused on the details during the majority of her design task:

She focused on fine-tuning the one solution by considering the material she would use and figuring out the detailed dimensions of each component. Since she only considered one solution, she did not engage in concept selection.

Student participants who generated multiple concepts lacked structures for how they selected the most promising idea. They showed favoritism in evaluating ideas and used inconsistent criteria to compare ideas. For example, student participants focused on their favorite idea and neglected other possible solutions:

Student participants used inconsistent evaluation criteria to compare ideas during Phase 1. For example, Henry was working on the personal transportation problem. Henry emphasized that his idea with treads would be good for recreation and quadcopter would be safer:

Henry did not use consistent criteria to compare all his ideas. He considered the quadcopter as a safe design, but he did not consider the safety of the tread idea. Although he listed some benefits of his ideas, he ultimately picked his tread idea for its convenience in personal use. Henry did not use a clear structure in his concept selection.

5.1. Participants

All student participants who completed the previous design task using their natural approaches were invited to complete the learning intervention, the Hybrid Learning Blocks. Ten mechanical engineering students, from Phase 1 completed them as well as a second design task and interview. Participants were compensated 200 USD for approximately 18 hours of their time.

Student participants were instructed to complete three Hybrid Learning Blocks created by the Center for Socially Engaged Design in the following sequence: “Idea Generation,” “Concept Development” and “Concept Selection” within a 3–4 week time frame (Center for Socially Engaged Design n.d.). Each learning block took approximately 5–7 hours to complete. Each block had specific learning objectives aligned with recommended practices in the particular idea phase. The “Idea Generation” block focused on applying divergent thinking to conduct idea generation and exploring the solution space using various ideation techniques. The “Concept Development” block emphasized iterating on the ideas from idea generation and drawing out novelty in design solutions. Last, the “Concept Selection” block’s learning outcomes include filtering potential solutions through objective comparisons against needs specifications and using recommended techniques to evaluate concepts.

Once the student participants completed the Hybrid Learning Blocks, they did a post-block design task. This time, they developed ideas for the problem statement that they had not completed during the Phase 1 design task. The Phase 2 study structure was identical to the Phase 1 structure, except the interview protocol included a few additional questions related to student participants’ Hybrid Learning Block experiences.

5.2. Data analysis

After student participants completed the Phase 2 design task, their behaviors were analyzed and built on the existing codebook from Table 2. Their behaviors were compared to student participant behaviors from Phase 1. Additionally, we examined the sketched and think-aloud data to measure outcomes of idea generation, development and selection for Phases 1 and 2, including the quantity of total ideas, variety of total ideas, quantity of ideas developed, number of criteria used in selection and prioritization of criteria in selection. The metrics are summarized in Table 3 and described in more detail in the following paragraphs.

Table 3. Student participants’ idea generation, development and selection outcomes and measurement methods

5.3. Quantity of total ideas generated in all idea phases

To measure quantity, we leveraged practices used in prior research (Shah et al. Reference Shah, Kulkarni and Vargas-Hernandez2000; Linsey et al. Reference Linsey, Green, Murphy, Wood and Markman2005). A single product solution was defined in two different ways: (1) student participants clearly indicated an idea by having a sketch with descriptions of an idea or (2) student participants only described an idea in words but the idea covered two or more functions of the design. When student participants came up with single components of ideas using idea generation techniques, we did not count them as individual ideas. For example, a student participant used the Mind Map to come up with various different ways to power a snow transporter such as wind power, motor and solar power, and we did not count these individual components as an idea. When the same student participant used two or more components from his Mind Map to build possible solutions and sketched out the details, we counted them as ideas. Inter-rater reliability between two coders using this approach was 94%. The coders discussed all discrepancies and reached a full agreement prior to finalizing the findings. After quantifying the total ideas generated, we conducted a paired t-test with the error rate of alpha = 0.05 using the participants who completed both phases.

5.4. Quantity of ideas developed

In quantifying the ideas developed, we followed the same procedure as quantifying the total number of ideas with additional criteria. Two coders only counted ideas that were explicitly indicated by student participants as (1) combining components of previous ideas, (2) building on previous ideas and (3) developing ideas after initial generation. The inter-rater reliability was 83% and the coders discussed all discrepancies until they reached full agreement prior to finalizing the findings. Both the think-aloud and interview data were used to analyze the quantity of ideas developed. After agreement with the coders, a paired t-test with the error rate of alpha = 0.05 was used for participants who completed both phases.

5.5. Variety of ideas

The variety of ideas was measured based on two different approaches: (1) ideas were grouped based on key features of the design to capture different types of ideas generated and (2) ideas were broken down into various functions or “bins” to analyze different subfunctions of ideas that student participants’ considered.

Each idea was classified by solution type based on the key features of the design, similar to approaches used in other studies measuring variety (Daly et al. Reference Daly, Christian, Yilmaz, Seifert and Gonzalez2012a; Jablokow et al. Reference Jablokow, Teerlink, Yilmaz, Daly, Silk and Wehr2015). For example, in the one-handed opener problem, all concepts that focused on using a handheld tool to pry open the container were classified as a type of solution that occurred several times among many participants. A coding scheme was created that consisted of exclusive categories differentiating “obvious” ideas from unexpected ideas. For the low-skill snow transporter problem, eight different codes were created (1-ATV, 2-snowmobile, 3-snowboard, 4-snowshoes, 5-ski, 6-scooter, 7-motorcycle, 8-other). For the one-handed opener problem, five different codes were created (1-base/lid restraint, 2-machine (twist), 3-handheld tool (puncture), 4-handheld tool (pry), 5-other). For both problems, the “other” category represented combinations of features from multiple categories and ideas that did not fit into the above categories. For example, in the low-skill snow transporter problem, any flying objects such as drones were placed in the “other” category. Using two coders, inter-rater reliability was 78%. The coders discussed all discrepancies and reached a full agreement prior to finalizing the findings.

For the second way of measuring the variety of ideas, we created categories based on various functions or “bins” of ideas (Linsey et al. Reference Linsey, Tseng, Fu, Cagan, Wood and Schunn2010). For example, in the low-skilled snow transporter problem, participants came up with a variety of ways to power their transporter including solar energy, wind energy, battery, magnetic force, and so forth. Each method of powering the snow transporter would be considered a bin. Based on all the bins, we counted how many bins were considered unique, meaning they were used by a limited number of student participants in this study. We counted bins that were only used by 1, 2 or 3 student participants. Then we compared how many of those unique bins were used by student participants during the Phase 1 and Phase 2 design tasks. Using two coders, inter-rater reliability was 71%. The coders discussed all discrepancies and reached full agreement prior to finalizing the findings. After coding, we used a paired t-test with only the participants who completed both phases with the error rate of alpha = 0.05 to conduct statistical analysis.

5.6. Number of criteria considered and prioritization of criteria in idea selection

Two coders counted the number of criteria that student participants considered during their idea selection and created a binary system: (1) an approach that used only one criterion in selecting ideas and (2) an approach that considered multiple criteria. Also, we examined if student participants prioritized their evaluation criteria in selecting ideas. Student participants who prioritized their criteria either ranked criteria or assigned different weighing values to each criterion to indicate their importance. We did not evaluate the specific idea selected by student participants because the focus of the work was to characterize students’ idea generation, development and selection processes.

6.1. Phase 2 idea generation approaches

After completion of the Hybrid Learning Blocks, student participants minimized assuming requirements in the early phase of idea generation, emphasized and generated unconventional solutions, utilized one or two idea generation tools and focused on increasing the quantity of ideas. The combination of these approaches led to student participants coming up with a larger quantity of diverse ideas.

While many student participants in Phase 1 generated assumed requirements, student participants in Phase 2 emphasized the importance of not limiting ideas early in idea generation and did not show signs of coming up with assumed requirements. For example:

Similar to the Phase 1 design task, student participants started idea generation with existing ideas, however, they then intentionally looked for unconventional ideas to help them diverge in idea generation. After coming up with several existing ideas, Cathy, who worked on the one-handed container opener problem, looked for unconventional ways to open a jar for her seventh idea:

Unlike Phase 1, student participants utilized at least one idea generation technique, including Design Heuristics, Mind Mapping, Morphological Matrix and SCAMPER. Using idea generation tools often helped student participants to approach idea generation in a structured way. For example, Brian was working on the snow transporter problem and used a Mind Map to generate ideas. His Mind Map incorporated central nodes that described the characteristics of his design such as power, snow movement, control direction and braking. Then he created components for each central node. For example, he thought of different ways to power a snow transporter such as wind power, solar power, turbine, jet snow propulsion, and so forth, as shown in Figure 2a.

Figure 2. (a) An example Mind Map used to generate ideas and (b) ideas synthesized from combining ideas in the Mind Map.

After coming up with various different functions within Mind Map, Brian combined multiple functions to create ideas. As seen in Figure 2b, one idea used wind power and a smooth surface to create a snow sail. The user can control direction by turning the sail and brake by moving the sail away from the wind. Another idea used jet propulsion and a smooth surface to create a snowmobile with a jet engine. By combining various functions from ideation techniques, student participants generated a number of different ideas.

In Phase 2, student participants articulated that coming up with a large quantity is important in idea generation. They focused on generating a lot of ideas that may be wild and unconventional, which is considered a recommended practice in idea generation. For example:

Student participants aimed to diverge to generate a large quantity of ideas and also gave themselves a target number of ideas to generate. In Phase 2, student participants articulated a clear quantity goal. For example:

By setting a clear quantity goal in idea generation, student participants generated a large number of ideas to ensure that they consider multiple ideas before evaluating them.

6.2. Phase 2 idea development approaches

In Phase 2, student participants intentionally developed ideas and they separated idea generation and development as two distinct phases in design. For example:

By articulating idea development as a phase, student participants set aside time to build on their previous ideas. Participants said that after coming up with initial ideas, they used idea development strategies such as Design Heuristics to help them build on their initial ideas. For example:

By having explicit idea generation and development phases, participants built on their initial ideas to ultimately have a larger quantity of ideas.

6.3. Phase 2 idea selection approaches

In the Phase 2 design task, student participants systematically organized their ideas into groups and used idea selection methods such as a decision matrix to select their final idea. After generating a large quantity of ideas, participants grouped their ideas based on similarities. For many student participants, the initial grouping of ideas helped them discard similar ideas before using a decision matrix. Afterward, student participants listed important criteria or requirements for their ideas. Then, student participants often assigned weighing values for each criterion with minimal justification. For example:

Henry came up with weighing values based on what was believed to be important.

After coming up with criteria and weighing values, student participants attempted to be objective in evaluating ideas based on how well each idea meets the criteria. Student participants compared ideas and depending on the comparison, they assigned appropriate values. For example:

Overall, in the Phase 2 design task, student participants attempted to limit their biases by using idea selection methods but their process was influenced by their perception of what criteria were important in their final idea. While they showed improvement in using a structured approach, student participants struggled with coming up with fair evaluation criteria.

7.1. Idea generation outcomes

There were some notable differences in the Phase 1 and 2 quantity of ideas. Student participants generated an average of 5.3 (SE 0.66) and 14.3 ideas (SE 1.47) in Phase 1 and 2, respectively, with a p-value of 0.00006 (Figure 3).

Figure 3. The average quantity of ideas generated in the design task during Phases 1 and 2.

With regards to differences in variety, for the first variety metric we applied – idea type – student participants generated fewer expected concept types in Phase 2 as compared to Phase 1. For the one-hand container opener problem, 70% of ideas in Phase 1 and 26% of ideas in Phase 2 involved either the base or lid being restrained before opening the container, representing the most obvious idea. The “other” category represents ideas that were not similar to other concepts in the pool of all concepts, making it a unique category itself. Thus, ideas in the “other” category did not fit into the other categories, comprising 13% in Phase 1 versus 63% in Phase 2 for the one-hand container opener problem. These comparisons are represented in Figure 4a. In the snow transporter problem, 38% and 50% of the ideas were categorized as “other” in Phases 1 and 2, respectively (represented in Figure 4b). Figure 4 demonstrates a visual representation of different types of ideas generated by the student participants to showcase the dominant and nondominant ideas considered.

Figure 4. (a) Percent of ideas grouped by solution types for Phases 1 and 2 for the one-handed container opener problem and (b) percent of ideas grouped by solution types for Phases 1 and 2 for the snow transporter problem.

We saw a similar trend of more unconventional concepts generated for Phase 2 when using the second variety metric we applied – the frequency of unusual features. By sorting ideas into bins of similar ideas, we found that on average, 0.7 (SE 0.27) and 1.2 (SE 0.46) bins were occupied by only one student participant in Phases 1 and 2, respectively, with a p-value of 0.45. This indicates that on average, student participants generated less than one (0.7) unique feature during Phase 1 while student participants came up with more than one (1.2) unique subfeature in Phase 2. On average, 1.2 (SE 0.25) and 2.5 (SE 0.5) bins were occupied by two or fewer student participants in Phase 1 and 2, respectively, with a p-value of 0.08. On average, 1.6 (SE 0.31) and 3.2 (SE 0.5) bins were occupied by three or fewer students in Phases 1 and 2, respectively, with a p-value of 0.03 (Figure 5); in other words, student participants in Phase 1 came up with on 1.6 features unique to three or fewer student participants while student participants in Phase 2 generated on average 3.2 subfeatures unique to three or fewer student participants. This analysis shows that students came up with more unique features of ideas in Phase 2.

Figure 5. Number of bins occupied by 1, 1–2 or 1–3 student participants.

7.2. Idea development outcomes

There were differences in the number of ideas developed. We counted concepts to be developed if student participants (1) combined components of previous ideas, (2) built on previous ideas and (3) came back to initial ideas to further develop them. In general, student participants in Phase 1 did not develop their initial ideas. Student participants developed an average of 0.7 (SE 0.2) and 4.6 concepts (SE 0.9) for Phases 1 and 2, respectively, with a p-value of 0.0006. These results are represented in Figure 6.

Figure 6. Quantity of developed ideas.

7.3. Idea selection outcomes

During Phase 1, student participants focused on a single idea throughout the task or showed minimal evaluation to select a single idea. For example, David was working on the one-hand container opener problem. He selected an idea based on practicality without considering or prioritizing other criteria:

After going through the learning blocks, student participants considered multiple, consistent criteria and compared their ideas before choosing their final one. Also, student participants prioritized multiple criteria by ranking criteria or providing a weighing value for each criterion. Student participants heavily relied on using a decision matrix to help them compare ideas and placed numerical values depending on their perceived quality of each idea. In the end, student participants added up all the values and became focused on picking the idea with the highest rating:

Although Cathy’s three ideas came out to be similar in the total value (12, 11 and 14 units) in her decision matrix, Cathy picked idea 3, which had 14 units without further questioning or reasoning her choice. Although student participants balanced benefits and trade-offs, they became fixated on picking the solution with the highest quantitative value, which may be a limiting behavior that does not align with recommended practices in idea selection.