Hostname: page-component-586b7cd67f-t7fkt Total loading time: 0 Render date: 2024-11-29T03:39:06.771Z Has data issue: false hasContentIssue false
  • English
  • Français

A Systematic Approach to Evaluating Design Prompts in Supporting Experimental Design Research

Part of: Methods of Design Research

Published online by Cambridge University Press:  26 July 2019

Apurva Patel ,
Maria-Vittoria Elena  and
Joshua Summers

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Experiments that study engineering behavior in design often rely on participants responding to a given design prompt or a problem statement. Moreover, researchers often find themselves testing multiple variables with a relatively small participant pool. In such situations multiple design prompts may be used to boost replication by giving each participant an equivalent problem with a different experimental condition. This paper presents a systematic approach to compare given design prompts using a two-step process that allows an initial comparison of the prompts and a post-experiment verification of the similarity of the given prompts. Comparison metrics are provided which can be used to evaluate a level of similarity of existing prompts as well as develop similar problems. These metrics include complexity (size, coupling, and solvability), familiarity, and prompt structure. Statistical methods are discussed for post-experiment verification. Guidelines are provided for a post-experiment survey which may be used for an additional perspective of prompt similarity. The proposed approach is demonstrated using an experiment where two design prompts were used for within-subject replication.

Keywords

Design theoryResearch methodologies and methodsEarly design phasesDesign problem
Type
Article
Information
Proceedings of the Design Society: International Conference on Engineering Design , Volume 1 , Issue 1 , July 2019 , pp. 2755 - 2764
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
© The Author(s) 2019

References

Cardoso, C. and Badke-Schaub, P. (2011), “The influence of different pictorial representations during idea generation”, Journal of Creative Behavior, Vol. 45 No. 2, pp. 130–146.Google Scholar
Chawla, A. and Summers, J.D. (2018), “Function Ordering Within Morphological Charts: An Experimental Study”, ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, pp. V007T06A012–V007T06A012.Google Scholar
Creswell, J. (2002), “Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research, Fourth”., Pearson Education.Google Scholar
Cross, N. (2004), “Expertise in design: An overview”, Design Studies, Vol. 25 No. 5, pp. 427–441.Google Scholar
Dorst, K. (2003), “The problem of design problems”, Expertise in Design, pp. 135–147.Google Scholar
Dorst, K. and Cross, N. (2001), “Creativity in the design process: Co-evolution of problem-solution”, Design Studies, Vol. 22 No. 5, pp. 425–437.Google Scholar
Durand, F., Helms, M.E., Tsenn, J., McAdams, D.A. and Linsey, J.S. (2015), “In Search of Effective Design Problems for Design Research”, Vol. 7: 27th International Conference on Design Theory and Methodology, ASME, p. V007T06A011.Google Scholar
Fu, K., Cagan, J. and Kotovsky, K. (2010), “Design Team Convergence: The Influence of Example Solution Quality”, Journal of Mechanical Design, Vol. 132 No. 11, p. 111005.Google Scholar
Goel, V. and Pirolli, P. (1992), “The structure of Design Problem Spaces”, Cognitive Science, Vol. 16 No. 3, pp. 395–429.Google Scholar
Goldschmidt, G. and Smolkov, M. (2006), “Variances in the impact of visual stimuli on design problem solving performance”, Design Studies, Vol. 27 No. 5, pp. 549–569.Google Scholar
Jansson, D.G. and Smith, S.M. (1991), “Design fixation”, Design Studies.Google Scholar
Joshi, S. and Summers, J.D. (2014), “Impact of Requirements Elicitation Activity on Idea Generation: A Designer Study”, Detc, No. November, pp. 1–10.Google Scholar
Kumar, V. and Mocko, G. (2016), “Similarity of Engineering Design Problems to Enable Reuse in Design Research Experiments”, Vol. 7: 28th International Conference on Design Theory and Methodology, ASME, p. V007T06A042.Google Scholar
Levy, B., Hilton, E., Tomko, M. and Linsey, J. (2017), “Investigating Problem Similarity Through Study of Between-Subject and Within-Subject Experiments”, Vol. 7: 29th International Conference on Design Theory and Methodology, ASME, p. V007T06A012.Google Scholar
Linsey, J.S., Clauss, E.F., Kurtoglu, T., Murphy, J.T., Wood, K.L. and Markman, A. B. (2011), “An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods”, Journal of Mechanical Design, Vol. 133 No. 3, p. 031008.Google Scholar
Linsey, J.S., Green, M.G., Murphy, J.T., Wood, K.L. and Markman, A.B. (2005), “‘Collaborating To Success’: An Experimental Study of Group Idea Generation Techniques”, Vol. 5a: 17th International Conference on Design Theory and Methodology, Vol. 2005, ASME, pp. 277–290.Google Scholar
Linsey, J.S., Wood, K.L. and Markman, A.B. (2008), “Increasing Innovation: Presentation and Evaluation of the Wordtree Design-by-Analogy Method”, Vol 4: 20th International Conference on Design Theory and Methodology; Second International Conference on Micro- and Nanosystems, ASME, pp. 21–32.Google Scholar
Mckoy, F.L., Vargas-hernández, N., Summers, J.D. and Shah, J.J. (2001), “Influence of design representation on effectiveness of idea generation”, Proceedings of the ASME Design Engineering Technical Conference, Pittsburgh, PA, United States, pp. 39–48.Google Scholar
Ozkan, O. and Dogan, F. (2013), “Cognitive strategies of analogical reasoning in design: Differences between expert and novice designers”, Design Studies, Elsevier, Vol. 34 No. 2, pp. 161–192.Google Scholar
Pahl, G., Beitz, W., Blessing, L., Feldhusen, J., Grote, K.-H.H. and Wallace, K. (2013), Engineering Design: A Systematic Approach, edited by Second, 3rd ed., Vol. 11, Springer-Verlag London Limited, London.Google Scholar
Patel, A.R., Summers, J.D., Kramer, W.S. and Shuffler-Porter, M.L. (2016), “Function modeling: A study of sequential model completion based on count and chaining of functions”, Proceedings of the ASME Design Engineering Technical Conference, Vol. 7, available at: https://doi.org/10.1115/DETC201659860.Google Scholar
Richardson, J.L. III, Summers, J.D. and Mocko, G.M. (2011), “Function Representations in Morphological Charts: An Experimental Study on Variety and Novelty on Means Generated”, Interdisciplinary Design: Proceedings of the 21st CIRP Design Conference, p. 76.Google Scholar
Sen, C. and Summers, J.D. (2012), “A Pilot Protocol Study on How Designers Construct Function Structures in Novel Design”, edited by Gero, J. 5th International Conference on Design Computing and Cognition, Springer-Verlag, College Station, TX, No. 37.Google Scholar
Shah, J.J., Smith, S.M. and Vargas-Hernández, N. (2003), “Metrics for Measuring Ideation Effectiveness”, Design Studies, Vol. 24, pp. 111–134.Google Scholar
Smith, G., Troy, T.J. and Summers, J.D. (2006), “Concept Exploration Through Morphological Charts: An Experimental Study”, Vol. 4a: 18th International Conference on Design Theory and Methodology, Vol. 2006, ASME, pp. 495–504.Google Scholar
Summers, J.D. and Shah, J.J. (2010), “Mechanical Engineering Design Complexity Metrics: Size, Coupling, and Solvability”, Journal of Mechanical Design, American Society of Mechanical Engineers, Vol. 132 No. 2, p. 021004.Google Scholar
Thoe, S. and Summers, J.D. (2013), “Correlating Problem/Process Exam Question Complexity to Anticipated Effort: A Modeling Protocol”, Vol. 1: 15th International Conference on Advanced Vehicle Technologies; 10th International Conference on Design Education; 7th International Conference on Micro- and Nanosystems, ASME, p. V001T04A012.Google Scholar
Ullman, D.G. (1992), “The Mechanical Design Process”, Maelabsucsdedu, Vol. 3. ed, McGraw-Hill, New York, NY.Google Scholar
Ulrich, K.T. and Eppinger, S.D. (2008), “Product Design and Development”, Product Design and Development, 4th ed., Vol. 384, McGraw-Hill, New York, USA, available at: https://doi.org/10.1016/B978-0-7506-8985-4.00002-4.Google Scholar
Wetmore, W.R., Summers, J.D. and Greenstein, J.S. (2010), “Experimental study of influence of group familiarity and information sharing on design review effectiveness”, Journal of Engineering Design, Vol. 21 No. 1, pp. 111–126.Google Scholar
Worinkeng, E., Joshi, S. and Summers, J.D. (2015), “An experimental study: analyzing requirement type influence on novelty and variety of generated solutions”, International Journal of Design Creativity and Innovation, Taylor & Francis, Vol. 3 No. 2, pp. 61–77.Google Scholar