Hostname: page-component-586b7cd67f-g8jcs Total loading time: 0 Render date: 2024-11-21T23:47:05.407Z Has data issue: false hasContentIssue false
  • English
  • Français

Replication studies in engineering design – a feasibility study

Published online by Cambridge University Press:  16 May 2024

Jonas Rode ,
Ingo Jonuschies ,
Sven Matthiesen  and
Kilian Gericke
Jonas Rode
Affiliation:
University of Rostock, Germany
Ingo Jonuschies*
Affiliation:
University of Rostock, Germany
Sven Matthiesen
Affiliation:
Karlsruhe Institute of Technology, Germany
Kilian Gericke
Affiliation:
University of Rostock, Germany
*
[email protected]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

This paper examines the replicability of studies in design research triggered by the replication crisis in psychology. It highlights the importance of replicating studies to ensure the robustness of research results and examines whether the description in a publication is sufficient to replicate. Therefore, the publication of a reference study was analysed and a replication study was conducted. The design of the replication study appears similar to the reference study, but the results differ. Possible reasons for the differences and implications for replication studies are discussed.

Keywords

research methodologies and methodsdesign methodspublication biasevaluationreliability
Type
Design Theory and Research Methods
Information
Proceedings of the Design Society , Volume 4: DESIGN 2024 , May 2024 , pp. 115 - 124
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2024.

References

Ahmed, A., Hurwitz, D., Gestson, S. and Brown, S. (2021), “Differences between Professionals and Students in Their Visual Attention on Multiple Representation Types While Solving an Open-Ended Engineering Design Problem”, Journal of Civil Engineering Education, Vol. 147 No. 3.CrossRefGoogle Scholar
Ahmed, S. (2007), “Empirical research in engineering practice”, J. of Design Research, Vol. 6 No. 3, p. 359.CrossRefGoogle Scholar
Bak, T.H. (2016), “Bilingualism and Executive Function”, Linguistic Approaches to Bilingualism, Vol. 6 No. 5, pp. 699–717.Google Scholar
Bi, Y., Shergadwala, M., Reid, T. and Panchal, J.H. (2015), “Understanding the Utilization of Information Stimuli in Design Decision Making Using Eye Gaze Data”, Proceedings of the ASME Design Engineering Technical Conference.CrossRefGoogle Scholar
Blessing, L.T. and Chakrabarti, A. (Eds.) (2009), DRM, a Design Research Methodology, Springer London, London.CrossRefGoogle Scholar
Cash, P., Daalhuizen, J. and Hekkert, P. (2023), “Evaluating the efficacy and effectiveness of design methods: A systematic review and assessment framework”, Design Studies, Vol. 88, p. 101204.CrossRefGoogle Scholar
Chen, R., Chen, Y. and Riyanto, Y.E. (2021), “Best practices in replication: a case study of common information in coordination games”, Experimental Economics, Vol. 24 No. 1, pp. 2–30.Google Scholar
Cockburn, A., Dragicevic, P., Besançon, L. and Gutwin, C. (2020), “Threats of a replication crisis in empirical computer science”, Communications of the ACM, Vol. 63 No. 8, pp. 70–79.CrossRefGoogle Scholar
Curtis, M.J. and Abernethy, D.R. (2015), “Replication - why we need to publish our findings”, Pharmacology research & perspectives, Vol. 3 No. 4, e00164.CrossRefGoogle ScholarPubMed
DFGDFG (2017), “Replizierbarkeit von Forschungsergebnissen - Eine Replizierbarkeit von Forschungsergebnissen. Eine Stellungnahme der Deutschen Forschungsgemeinschaft”, available at: https://www.dfg.de/foerderung/info_wissenschaft/2017/info_wissenschaft_17_18/ (accessed 9 October 2023).Google Scholar
Du, P. and MacDonald, E.F. (2014), “Eye-Tracking Data Predict Importance of Product Features and Saliency of Size Change”, Journal of Mechanical Design, Vol. 136 No. 8.Google Scholar
Ericsson, K.A. and Simon, H.A. Ericsson, K. Anders; Simon, Herbert A. (1993), Protocol analysis: Verbal reports as data, Revised edition, [publisher not identified], [Place of publication not identified].CrossRefGoogle Scholar
Errington, T.M., et al. (2014), “An open investigation of the reproducibility of cancer biology research”, eLife, Vol. 3.CrossRefGoogle ScholarPubMed
Fanelli, D. (2010), “"Positive" results increase down the Hierarchy of the Sciences”, PloS one, Vol. 5 No. 4, e10068.CrossRefGoogle ScholarPubMed
Ferguson, C.J. and Heene, M. (2012), “A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null”, Perspectives on psychological science a journal of the Association for Psychological Science, Vol. 7 No. 6, pp. 555–561.Google Scholar
Gericke, K., Eckert, C. and Stacey, M. (2022), “Elements of a design method – a basis for describing and evaluating design methods”, Design Science, Vol. 8.Google Scholar
Hussy, W., Schreier, M. and Echterhoff, G.Hussy, Walter; Schreier, Margrit; Echterhoff, Gerald (2013), Forschungsmethoden in Psychologie und Sozialwissenschaften für Bachelor, Springer Berlin Heidelberg, Berlin, Heidelberg.Google Scholar
Ioannidis, J.P.A. (2005), “Why most published research findings are false”, PLoS medicine, Vol. 2 No. 8, e124.Google ScholarPubMed
John, L.K., Loewenstein, G. and Prelec, D. (2012), “Measuring the prevalence of questionable research practices with incentives for truth telling”, Psychological science, Vol. 23 No. 5, pp. 524–532.CrossRefGoogle ScholarPubMed
Kelley, T.R., Capobianco, B.M. and Kaluf, K.J. (2015), “Concurrent think-aloud protocols to assess elementary design students”, International Journal of Technology and Design Education, Vol. 25 No. 4, pp. 521–540.CrossRefGoogle Scholar
Kelter, R. (2020), “Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research”, BMC medical research methodology, Vol. 20 No. 1, p. 88.CrossRefGoogle ScholarPubMed
Klein, R.A., et al. (2018), “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings”, Advances in Methods and Practices in Psychological Science, Vol. 1 No. 4, pp. 443–490.CrossRefGoogle Scholar
Klein, R.A., et al. (2022), “Many Labs 4: Failure to replicate mortality salience effect with and without original author involvement”, Collabra: Psychology, Vol. 8 No. 1, p. 35271.CrossRefGoogle Scholar
Makel, M.C., Plucker, J.A. and Hegarty, B. (2012), “Replications in Psychology Research: How Often Do They Really Occur?”, Perspectives on psychological science a journal of the Association for Psychological Science, Vol. 7 No. 6, pp. 537–542.Google ScholarPubMed
Matthiesen, S., Meboldt, M., Ruckpaul, A. and Mussgnug, M. (2013), “Eye Tracking, a method for engineering design research on engineers's behavior while analyzing technical systems”, ICED 13.Google Scholar
Matthiesen, S. and Nelius, T. (2018), “Managing Assumptions during Analysis - Study on successful Approaches of Design Engineers”, paper presented at NordDesign 2018.Google Scholar
Nissen, S.B., Magidson, T., Gross, K. and Bergstrom, C.T. (2016), “Publication bias and the canonization of false facts”, eLife, Vol. 5.CrossRefGoogle ScholarPubMed
Collaboration, Open Science (2015), “Estimating the reproducibility of psychological science”, Science (New York, N.Y.), Vol. 349 No. 6251, aac4716.CrossRefGoogle Scholar
Pashler, H. and Wagenmakers, E.-J. (2012), “Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?”, Perspectives on psychological science a journal of the Association for Psychological Science, Vol. 7 No. 6, pp. 528–530.Google Scholar
Pavlov, Y.G., et al. (2021), “#EEGManyLabs: Investigating the replicability of influential EEG experiments”, Cortex; a journal devoted to the study of the nervous system and behavior, Vol. 144, pp. 213–229.CrossRefGoogle ScholarPubMed
Peels, R. and Bouter, L. (2018), “The possibility and desirability of replication in the humanities”, Palgrave Communications, Vol. 4 No. 1.CrossRefGoogle Scholar
Pridemore, W.A., Makel, M.C. and Plucker, J.A. (2018), “Replication in Criminology and the Social Sciences”, Annual Review of Criminology, Vol. 1 No. 1, pp. 19–38.Google Scholar
Roopa, S. and Rani, M.S. (2012), “Questionnaire Designing for a Survey”, The Journal of Indian Orthodontic Society, Vol. 46, pp. 273–277.CrossRefGoogle Scholar
Rougier, N.P., et al. (2017), “Sustainable computational science: the ReScience initiative”, PeerJ. Computer science, Vol. 3, e142.CrossRefGoogle ScholarPubMed
Ryan, J.C. and Tipu, A, S.A. (2022), “Business and management research: Low instances of replication studies and a lack of author independence in replications”, Research Policy, Vol. 51 No. 1, p. 104408.CrossRefGoogle Scholar
Stroebe, W. (2016), “Are most published social psychological findings false?”, Journal of Experimental Social Psychology, Vol. 66, pp. 134–144.CrossRefGoogle Scholar
Ulrich, R., et al. (2016), “Inflation von falsch-positiven Befunden in der psychologischen Forschung”, Psychologische Rundschau, Vol. 67 No. 3, pp. 163–174.Google Scholar
Wallace, K. (2011), “Transferring Design Methods into Practice”, in Birkhofer, H. (Ed.), The future of design methodology, Springer, London, New York, pp. 239–248.CrossRefGoogle Scholar