Hostname: page-component-78c5997874-fbnjt Total loading time: 0 Render date: 2024-11-19T10:09:48.108Z Has data issue: false hasContentIssue false
  • English
  • Français

PEVALUATING ENGINEERING DESIGN METHODS: TAKING INSPIRATION FROM SOFTWARE ENGINEERING AND THE HEALTH SCIENCES

Part of: Socio-technical Issues in Design

Published online by Cambridge University Press:  11 June 2020

A. M. Hein  and
G. Lamé
A. M. Hein*
Affiliation:
CentraleSupélec, France
G. Lamé
Affiliation:
CentraleSupélec, France
*
*[email protected]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Engineering design methods are typically evaluated via case studies, surveys, and experiments. Meanwhile, domains such as the health sciences as well as software engineering have developed further powerful evaluation approaches. The objective of this paper is to show how evaluation approaches from the health sciences and software engineering might further the evaluation of engineering design methods. We survey these approaches and show which approaches could be transferred to the evaluation of engineering design methods.

Keywords

design methodsdesign methodologyevaluationempirical studiesresearch methodologies and methods
Type
Article
Information
Proceedings of the Design Society: DESIGN Conference , Volume 1 , May 2020 , pp. 1901 - 1910
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2020. Published by Cambridge University Press

References

Badke-Schaub, P. and Frankenberger, E. (1999), “Analysis of design projects”, Design Studies, Elsevier, Vol. 20 No. 5, pp. 465–480.CrossRefGoogle Scholar
Barth, A., Caillaud, E. and Rose, B. (2011), “How to validate research in engineering design?”, ICED 2011, Vol. 2: De, The Design Society, Lyngby/Copenhagen.Google Scholar
Blessing, L. and Chakrabarti, A. (2009), DRM, a Design Research Methodology, Springer, London.10.1007/978-1-84882-587-1CrossRefGoogle Scholar
Breuer, E. et al. (2016), “Using theory of change to design and evaluate public health interventions: a systematic review”, Implementation Science : IS, Vol. 11, p. 63.CrossRefGoogle ScholarPubMed
Bryant, C.R. et al. (2006), “A validation study of an automated concept generator design tool”, ASME 2006 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 283–294.10.1115/DETC2006-99489CrossRefGoogle Scholar
Bunning, C. (1995), Professional Development Using Action Research. Look Forward Ask Questions Learn Virtual Conference, MCB University Press.Google Scholar
Cash, P.J. (2018), “Developing theory-driven design research”, Design Studies, Vol. 56, pp. 84–119.10.1016/j.destud.2018.03.002CrossRefGoogle Scholar
Craig, P. et al. (2008), “Developing and evaluating complex interventions: the new Medical Research Council guidance”, BMJ (Clinical Research Ed.), Vol. 337, pp. a1655–a1655.Google ScholarPubMed
Cristiano, J.J., Liker, J.K. and White, C.I. (2001), “Key factors in the successful application of quality function deployment (QFD)”, IEEE Transactions on Engineering Management, Vol. 48 No. 1, pp. 81–95.CrossRefGoogle Scholar
Dingsøyr, T., Dybå, T. and Abrahamsson, P. (2008), “A preliminary roadmap for empirical research on agile software development”, Agile 2008 Conference, IEEE, pp. 83–94.10.1109/Agile.2008.50CrossRefGoogle Scholar
Dorst, K. (2008), “Design research: a revolution-waiting-to-happen”, Design Studies, Elsevier, Vol. 29 No. 1, pp. 4–11.CrossRefGoogle Scholar
Dybå, T. and Dingsøyr, T. (2008), “Strength of evidence in systematic reviews in software engineering”, Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ACM, pp. 178–187.10.1145/1414004.1414034CrossRefGoogle Scholar
Dybå, T., Sjøberg, D.I. and Cruzes, D.S. (2012), “What works for whom, where, when, and why?: on the role of context in empirical software engineering”, Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ACM, pp. 19–28.CrossRefGoogle Scholar
Easterbrook, S. et al. (2008), “Selecting empirical methods for software engineering research”, Guide to Advanced Empirical Software Engineering, Springer, pp. 285–311.10.1007/978-1-84800-044-5_11CrossRefGoogle Scholar
Fenton, N. (2001), “Viewpoint Article: Conducting and presenting empirical software engineering”, Empirical Software Engineering, Vol. 6 No. 3, pp. 195–200.CrossRefGoogle Scholar
Ferreira, I.M. and Gil, P.J. (2012), “Application and performance analysis of neural networks for decision support in conceptual design”, Expert Systems with Applications, Vol. 39 No. 9, pp. 7701–7708.CrossRefGoogle Scholar
Frey, D.D. and Dym, C.L. (2006), “Validation of design methods: lessons from medicine”, Research in Engineering Design, Vol. 17 No. 1, pp. 45–57.CrossRefGoogle Scholar
Gericke, K., Eckert, C.M. and Stacey, M. (2017), “What do we need to say about a design method?”, ICED 2017, Vol. 7: De, The Design Society, Vancouver, BC, Canada.Google Scholar
Green, J. and Britten, N. (1998), “Qualitative research and evidence based medicine”, BMJ, Vol. 316 No. 7139, pp. 1230–1232.10.1136/bmj.316.7139.1230CrossRefGoogle ScholarPubMed
Griffin, A. (1991), Evaluating Development Processes: QFD as an Example.Google Scholar
Hannay, J.E. et al. (2009), “The effectiveness of pair programming: A meta-analysis”, Information and Software Technology, Vol. 51 No. 7, pp. 1110–1122.10.1016/j.infsof.2009.02.001CrossRefGoogle Scholar
Hawe, P., Shiell, A. and Riley, T. (2009), “Theorising interventions as events in systems”, American Journal of Community Psychology, Vol. 43 No. 3–4, pp. 267–276.10.1007/s10464-009-9229-9CrossRefGoogle ScholarPubMed
Hedges, L.V. and Olkin, I. (2014), Statistical Methods for Meta-Analysis, Academic press.Google Scholar
Hein, A. (2016), Heritage Technologies in Space Programs - Assessment Methodology and Statistical Analysis, PhD thesis, Technical University of Munich.Google Scholar
Kitchenham, B.A. et al. (2002), “Preliminary guidelines for empirical research in software engineering”, IEEE Transactions on Software Engineering, Vol. 28 No. 8, pp. 721–734.10.1109/TSE.2002.1027796CrossRefGoogle Scholar
Ko, A.J., Latoza, T.D. and Burnett, M.M. (2015), “A practical guide to controlled experiments of software engineering tools with human participants”, Empirical Software Engineering, Vol. 20 No. 1, pp. 110–141.10.1007/s10664-013-9279-3CrossRefGoogle Scholar
Lamé, G. (2019), “Systematic literature reviews: an introduction”, ICED19 - 22nd International Conference on Engineering Design, Design Society.Google Scholar
Lamé, G. and Dixon-Woods, M. (n.d.). “Using clinical simulation to study how to improve quality and safety in healthcare”, BMJ Simulation and Technology Enhanced Learning, available at: https://doi.org/10.1136/bmjstel-2018-000370.Google Scholar
Merlin, T., Weston, A. and Tooher, R. (2009), “Extending an evidence hierarchy to include topics other than treatment: revising the Australian ‘levels of evidence’”, BMC Medical Research Methodology, Vol. 9 No. 1, p. 34.10.1186/1471-2288-9-34CrossRefGoogle ScholarPubMed
Minary, L. et al. (2019), “Which design to evaluate complex interventions? Toward a methodological framework through a systematic review”, BMC Medical Research Methodology, Vol. 19 No. 1, p. 92.CrossRefGoogle Scholar
Moore, G.F. et al. (2015), “Process evaluation of complex interventions: Medical Research Council guidance”, BMJ, Vol. 350, available at: https://doi.org/10.1136/bmj.h1258.CrossRefGoogle ScholarPubMed
Murad, M.H. et al. (2016), “New evidence pyramid”, Evidence-Based Medicine, Vol. 21 No. 4, pp. 125–127.10.1136/ebmed-2016-110401CrossRefGoogle ScholarPubMed
Perry, D.E., Porter, A.A. and Votta, L.G. (2000), “Empirical studies of software engineering: a roadmap”, Proceedings of the Conference on The Future of Software Engineering, ACM, pp. 345–355.CrossRefGoogle Scholar
Pickard, L.M., Kitchenham, B.A. and Jones, P.W. (1998), “Combining empirical results in software engineering”, Information and Software Technology, Vol. 40 No. 14, pp. 811–821.10.1016/S0950-5849(98)00101-3CrossRefGoogle Scholar
Radcliffe, D. and Harrison, P. (1994), Transforming Design Practice in a Small Manufacturing Enterprise, American Society of Mechanical Engineers, Design Engineering Division, Design Engineering Division (Publication) DE.Google Scholar
Rafique, Y. and Mišić, V.B. (2012), “The effects of test-driven development on external quality and productivity: A meta-analysis”, IEEE Transactions on Software Engineering, Vol. 39 No. 6, pp. 835–856.CrossRefGoogle Scholar
Runeson, P. et al. (2012), Case Study Research in Software Engineering: Guidelines and Examples, John Wiley & Sons.CrossRefGoogle Scholar
Seepersad, C.C. et al. (2006), “The Validation Square: How Does One Verify and Validate a Design Method?”, in Lewis, K.E., Chen, W. and Schmidt, L.C. (Eds.), Decision Making in Engineering Design, ASME, New York, NY, available at: http://doi.org/10.1115/1.802469.ch25.Google Scholar
Shi, Y.L.Z. et al. (2019), “Cognitive Style and Field Knowledge in Complex Design Problem-Solving: A Comparative Case Study of Decision Support Systems”, Design Computing and Cognition ’18, Springer International Publishing, pp. 341–360.10.1007/978-3-030-05363-5_19CrossRefGoogle Scholar
Shull, F., Singer, J. and Sjøberg, D.I. (2007), Guide to Advanced Empirical Software Engineering, Springer Science & Business Media.Google Scholar
De Silva, M.J. et al. (2014), “Theory of Change: a theory-driven approach to enhance the Medical Research Council's framework for complex interventions”, Trials, Vol. 15 No. 1, p. 267.10.1186/1745-6215-15-267CrossRefGoogle ScholarPubMed
Sio, U.N., Kotovsky, K. and Cagan, J. (2015), “Fixation or inspiration? A meta-analytic review of the role of examples on design processes”, Design Studies, Vol. 39, pp. 70–99.10.1016/j.destud.2015.04.004CrossRefGoogle Scholar
Sjoberg, D.I., Dyba, T. and Jorgensen, M. (2007), “The future of empirical methods in software engineering research”, 2007 Future of Software Engineering, IEEE Computer Society, pp. 358–378.Google Scholar