Hostname: page-component-586b7cd67f-l7hp2 Total loading time: 0 Render date: 2024-11-22T13:12:54.360Z Has data issue: false hasContentIssue false
  • English
  • Français

How good is ChatGPT? An exploratory study on ChatGPT's performance in engineering design tasks and subjective decision-making

Published online by Cambridge University Press:  16 May 2024

Wanyu Xu ,
Maulik Chhabilkumar Kotecha  and
Daniel A. McAdams
Wanyu Xu*
Affiliation:
Texas A&M University, United States of America
Maulik Chhabilkumar Kotecha
Affiliation:
Texas A&M University, United States of America
Daniel A. McAdams
Affiliation:
Texas A&M University, United States of America
*
*[email protected]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

This study explores how large language models like ChatGPT comprehend language and assess information. Through two experiments, we compare ChatGPT's performance with humans', addressing two key questions: 1) How does ChatGPT compare with human raters in evaluating judgment-based tasks like speculative technology realization? 2) How well does ChatGPT extract technical knowledge from non-technical content, such as mining speculative technologies from text, compared to humans? Results suggest ChatGPT's promise in knowledge extraction but also reveal a disparity with humans in decision-making.

Keywords

artificial intelligence (AI)design knowledgedesign researchdesign theory
Type
Artificial Intelligence and Data-Driven Design
Information
Proceedings of the Design Society , Volume 4: DESIGN 2024 , May 2024 , pp. 2307 - 2316
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2024.

References

Bleecker, J. (2009) Design Fiction: A Short Essay on Design, Science, Fact and Fiction, Near Future Laboratory, available: https://blog.nearfuturelaboratory.com/2009/03/17/design-fiction-a-short-essay-on-design-science-fact-and-fiction/.Google Scholar
Callaghan, V. (2015) “Creative Science Injecting Innovation into the IT Industry”, ITNOW, 57(2), 52–55.CrossRefGoogle Scholar
Christensen, B. (2019) Technovelgy.com, available: http://www.technovelgy.com/ [accessed September 11].Google Scholar
Dunne, A. and Raby, F. (2013) Speculative Everything: Design, fiction, and Social Dreaming, The MIT Press.Google Scholar
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Diego, Lisa, Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., George, Damoc, Guy, B., Osindero, A., Simonyan, S., Elsen, K., Jack, E., Vinyals, O. and Sifre, L. (2022) “Training Compute-Optimal Large Language Models“, arXiv pre-print server, available: http://doi.org/10.48550/arXiv:2203.15556.CrossRefGoogle Scholar
Ji, Y., Gong, Y., Peng, Y., Ni, C., Sun, P., Pan, D., Ma, B. and Li, X. (2023) “Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences”, arXiv pre-print server, available: http://doi.org/10.48550/arXiv:2303.07610.CrossRefGoogle Scholar
Kotecha, M.C., Chen, T.-J., McAdams, D.A. and Krishnamurthy, V. (2021) “Design Ideation Through Speculative Fiction: Foundational Principles & Exploratory Study”, Journal of Mechanical Design, 1–39, available: http://dx.doi.org/10.1115/1.4049656.CrossRefGoogle Scholar
Koubaa, A. (2023) “GPT-4 vs. GPT-3.5: A concise showdown”.CrossRefGoogle Scholar
Landis, J.R. and Koch, G.G. (1977) “The measurement of observer agreement for categorical data”, Biometrics, 33(1), 159–174, available: http://dx.doi.org/10.2307/2529310.CrossRefGoogle ScholarPubMed
Ma, K., Grandi, D., McComb, C. and Goucher-Lambert, K. (2023) “Conceptual Design Generation Using Large Language Models”, in ASME 2023 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, V006T06A021, available: http://dx.doi.org/10.1115/detc2023-116838.CrossRefGoogle Scholar
OpenAI (2022) Introducing ChatGPT, available: https://openai.com/blog/chatgpt [accessed November 13].Google Scholar
Ray, P.P. (2023) “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope”, Internet of Things and Cyber-Physical Systems, 3, 121–154, available: https://doi.org/10.1016/j.iotcps.2023.04.003.CrossRefGoogle Scholar
Siddharth, L., Blessing, L. and Luo, J. (2021) “Natural language processing in-and-for design research”, Design Science, 8.Google Scholar
Sterling, B. (2005) Shaping things, Cambridge, Massachusetts: The MIT Press.Google Scholar
Tom, Mann, Ryder, B., Subbiah, N., Kaplan, M., Dhariwal, J., Neelakantan, P., Shyam, A., Sastry, P., Askell, G., Agarwal, A., Herbert-Voss, S., Krueger, A., Henighan, G., Child, T., Ramesh, R., Daniel, A., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. (2020) “Language Models are Few-Shot Learners”, arXiv pre-print server, available: http://doi.org/10.48550/arxiv:2005.14165.CrossRefGoogle Scholar
Wang, B., Zuo, H., Cai, Z., Yin, Y., Childs, P., Sun, L. and Chen, L. (2023) “A Task-Decomposed AI-Aided Approach for Generative Conceptual Design”, in ASME 2023 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, V006T06A009, available: http://dx.doi.org/10.1115/detc2023-109087.CrossRefGoogle Scholar
Xu, W., Kotecha, M.C., Padilla, D., Jimenez, J. and McAdams, D.A. (2021) “Quantifying the Predictive Abilities of Speculative Fiction: A Feasibility Study”, in ASME 2021 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, V006T06A005, available: http://dx.doi.org/10.1115/detc2021-68723.CrossRefGoogle Scholar
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Xi, Mihaylov, Ott, T., Shleifer, M., Shuster, S., Simig, K., Punit, D., Sridhar, A., Wang, T. and Zettlemoyer, L. (2022) “OPT: Open Pre-trained Transformer Language Models”, arXiv pre-print server, available: http://dx.doi.org/10.48550/arXiv.2205.01068.CrossRefGoogle Scholar
Zhu, Q. and Luo, J. (2023) “Generative Transformers for Design Concept Generation”, Journal of Computing and Information Science in Engineering, 23(4), available: http://dx.doi.org/10.1115/1.4056220.Google Scholar
Zhu, Q., Zhang, X. and Luo, J. (2023) “Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers”, Journal of Mechanical Design, 145(4), available: http://dx.doi.org/10.1115/1.4056598.CrossRefGoogle Scholar
Zhu, Y., Zhang, P., Haq, E.-U., Hui, P. and Tyson, G. (2023) “Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks”, arXiv pre-print server, available: http://dx.doi.org/10.48550/arXiv.2304.10145.CrossRefGoogle Scholar