ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity

ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub,...

Full description

Saved in:
Bibliographic Details
Published in:European journal of engineering education 2023-07, Vol.48 (4), p.559-614
Main Authors: Nikolic, Sasha, Daniel, Scott, Haque, Rezwanul, Belkina, Marina, Hassan, Ghulam M., Grundy, Sarah, Lyden, Sarah, Neal, Peter, Sandison, Caz
Format: Article
Language:English
Subjects:
Artificial Intelligence
Artificial intelligence (AI)
assessment
Benchmarking
Chatbots
ChatGPT
Colleges & universities
Computer Assisted Testing
Engineering Education
Evaluation Methods
Foreign Countries
Generative artificial intelligence
GPT-3
Integrity
Interdisciplinary Approach
Performance Based Assessment
Prompting
Shock waves
Thinking Skills
Undergraduate Students
Universities
Writing Evaluation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573
cites cdi_FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573
container_end_page 614
container_issue 4
container_start_page 559
container_title European journal of engineering education
container_volume 48
creator Nikolic, Sasha
Daniel, Scott
Haque, Rezwanul
Belkina, Marina
Hassan, Ghulam M.
Grundy, Sarah
Lyden, Sarah
Neal, Peter
Sandison, Caz
description ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT's performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.
doi_str_mv 10.1080/03043797.2023.2213169
format article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2827824019</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ericid>EJ1390178</ericid><sourcerecordid>2827824019</sourcerecordid><originalsourceid>FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573</originalsourceid><addsrcrecordid>eNp9Uc1u1DAQjhBILKWPUMkS5yy2YycOJ9CqlKJK7aE9W44zzrpk7cV2tsor8lQ4TUGcuHik-X5mxl9RXBC8JVjgj7jCrGraZksxrbaUkorU7atiQ1jdlly04nWxWTjlQnpbvIvxEWNCOeeb4tdur9LV3T06QYhTROAG6wCCdQOCftIqWe-QihFiPIBLn5BCh2lMtrdR2-NonQozUq5fu6V1Mdk0LSo1og6c3h9U-LHYLSSVu3O0EXmD0j7XARyEPOQESIVkjdU266xLMI42gxpQ8n7MT26eIJsPKsE_Cz1zh2DT_L54Y9QY4fylnhUPXy_vd9_Km9ur692Xm1Iz3KSyFRrAtNyYhnWCdLjjXAvGayyA1qbjLa841bwDEBkwtWHQ1TUwJlTPeFOdFR9W32PwP6e8knz0U8iHRUkFbQRlmLSZxVeWDj7GAEYeg81fMUuC5RKb_BObXGKTL7Fl3cWqyxnov5rL76RqMWlExj-vuHXGh4N68mHsZVLz6IMJymkbZfX_Eb8BFFOv2Q</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2827824019</pqid></control><display><type>article</type><title>ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity</title><source>Taylor and Francis Science and Technology Collection</source><source>ERIC</source><creator>Nikolic, Sasha ; Daniel, Scott ; Haque, Rezwanul ; Belkina, Marina ; Hassan, Ghulam M. ; Grundy, Sarah ; Lyden, Sarah ; Neal, Peter ; Sandison, Caz</creator><creatorcontrib>Nikolic, Sasha ; Daniel, Scott ; Haque, Rezwanul ; Belkina, Marina ; Hassan, Ghulam M. ; Grundy, Sarah ; Lyden, Sarah ; Neal, Peter ; Sandison, Caz</creatorcontrib><description>ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT's performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.</description><identifier>ISSN: 0304-3797</identifier><identifier>EISSN: 1469-5898</identifier><identifier>DOI: 10.1080/03043797.2023.2213169</identifier><language>eng</language><publisher>Abingdon: Taylor &amp; Francis</publisher><subject>Artificial Intelligence ; Artificial intelligence (AI) ; assessment ; Benchmarking ; Chatbots ; ChatGPT ; Colleges &amp; universities ; Computer Assisted Testing ; Engineering Education ; Evaluation Methods ; Foreign Countries ; Generative artificial intelligence ; GPT-3 ; Integrity ; Interdisciplinary Approach ; Performance Based Assessment ; Prompting ; Shock waves ; Thinking Skills ; Undergraduate Students ; Universities ; Writing Evaluation</subject><ispartof>European journal of engineering education, 2023-07, Vol.48 (4), p.559-614</ispartof><rights>2023 The Author(s). Published by Informa UK Limited, trading as Taylor &amp; Francis Group 2023</rights><rights>2023 The Author(s). Published by Informa UK Limited, trading as Taylor &amp; Francis Group. This work is licensed under the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573</citedby><cites>FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573</cites><orcidid>0000-0002-5364-6011 ; 0000-0002-7528-9713 ; 0000-0002-5475-9640 ; 0000-0002-8831-5327 ; 0000-0002-8641-4479 ; 0000-0002-3305-9493 ; 0000-0002-6636-8807</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27924,27925</link.rule.ids><backlink>$$Uhttp://eric.ed.gov/ERICWebPortal/detail?accno=EJ1390178$$DView record in ERIC$$Hfree_for_read</backlink></links><search><creatorcontrib>Nikolic, Sasha</creatorcontrib><creatorcontrib>Daniel, Scott</creatorcontrib><creatorcontrib>Haque, Rezwanul</creatorcontrib><creatorcontrib>Belkina, Marina</creatorcontrib><creatorcontrib>Hassan, Ghulam M.</creatorcontrib><creatorcontrib>Grundy, Sarah</creatorcontrib><creatorcontrib>Lyden, Sarah</creatorcontrib><creatorcontrib>Neal, Peter</creatorcontrib><creatorcontrib>Sandison, Caz</creatorcontrib><title>ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity</title><title>European journal of engineering education</title><description>ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT's performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.</description><subject>Artificial Intelligence</subject><subject>Artificial intelligence (AI)</subject><subject>assessment</subject><subject>Benchmarking</subject><subject>Chatbots</subject><subject>ChatGPT</subject><subject>Colleges &amp; universities</subject><subject>Computer Assisted Testing</subject><subject>Engineering Education</subject><subject>Evaluation Methods</subject><subject>Foreign Countries</subject><subject>Generative artificial intelligence</subject><subject>GPT-3</subject><subject>Integrity</subject><subject>Interdisciplinary Approach</subject><subject>Performance Based Assessment</subject><subject>Prompting</subject><subject>Shock waves</subject><subject>Thinking Skills</subject><subject>Undergraduate Students</subject><subject>Universities</subject><subject>Writing Evaluation</subject><issn>0304-3797</issn><issn>1469-5898</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>0YH</sourceid><sourceid>7SW</sourceid><recordid>eNp9Uc1u1DAQjhBILKWPUMkS5yy2YycOJ9CqlKJK7aE9W44zzrpk7cV2tsor8lQ4TUGcuHik-X5mxl9RXBC8JVjgj7jCrGraZksxrbaUkorU7atiQ1jdlly04nWxWTjlQnpbvIvxEWNCOeeb4tdur9LV3T06QYhTROAG6wCCdQOCftIqWe-QihFiPIBLn5BCh2lMtrdR2-NonQozUq5fu6V1Mdk0LSo1og6c3h9U-LHYLSSVu3O0EXmD0j7XARyEPOQESIVkjdU266xLMI42gxpQ8n7MT26eIJsPKsE_Cz1zh2DT_L54Y9QY4fylnhUPXy_vd9_Km9ur692Xm1Iz3KSyFRrAtNyYhnWCdLjjXAvGayyA1qbjLa841bwDEBkwtWHQ1TUwJlTPeFOdFR9W32PwP6e8knz0U8iHRUkFbQRlmLSZxVeWDj7GAEYeg81fMUuC5RKb_BObXGKTL7Fl3cWqyxnov5rL76RqMWlExj-vuHXGh4N68mHsZVLz6IMJymkbZfX_Eb8BFFOv2Q</recordid><startdate>20230704</startdate><enddate>20230704</enddate><creator>Nikolic, Sasha</creator><creator>Daniel, Scott</creator><creator>Haque, Rezwanul</creator><creator>Belkina, Marina</creator><creator>Hassan, Ghulam M.</creator><creator>Grundy, Sarah</creator><creator>Lyden, Sarah</creator><creator>Neal, Peter</creator><creator>Sandison, Caz</creator><general>Taylor &amp; Francis</general><general>Taylor &amp; Francis Ltd</general><scope>0YH</scope><scope>7SW</scope><scope>BJH</scope><scope>BNH</scope><scope>BNI</scope><scope>BNJ</scope><scope>BNO</scope><scope>ERI</scope><scope>PET</scope><scope>REK</scope><scope>WWN</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7TA</scope><scope>7TB</scope><scope>8FD</scope><scope>FR3</scope><scope>JG9</scope><scope>KR7</scope><orcidid>https://orcid.org/0000-0002-5364-6011</orcidid><orcidid>https://orcid.org/0000-0002-7528-9713</orcidid><orcidid>https://orcid.org/0000-0002-5475-9640</orcidid><orcidid>https://orcid.org/0000-0002-8831-5327</orcidid><orcidid>https://orcid.org/0000-0002-8641-4479</orcidid><orcidid>https://orcid.org/0000-0002-3305-9493</orcidid><orcidid>https://orcid.org/0000-0002-6636-8807</orcidid></search><sort><creationdate>20230704</creationdate><title>ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity</title><author>Nikolic, Sasha ; Daniel, Scott ; Haque, Rezwanul ; Belkina, Marina ; Hassan, Ghulam M. ; Grundy, Sarah ; Lyden, Sarah ; Neal, Peter ; Sandison, Caz</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Artificial Intelligence</topic><topic>Artificial intelligence (AI)</topic><topic>assessment</topic><topic>Benchmarking</topic><topic>Chatbots</topic><topic>ChatGPT</topic><topic>Colleges &amp; universities</topic><topic>Computer Assisted Testing</topic><topic>Engineering Education</topic><topic>Evaluation Methods</topic><topic>Foreign Countries</topic><topic>Generative artificial intelligence</topic><topic>GPT-3</topic><topic>Integrity</topic><topic>Interdisciplinary Approach</topic><topic>Performance Based Assessment</topic><topic>Prompting</topic><topic>Shock waves</topic><topic>Thinking Skills</topic><topic>Undergraduate Students</topic><topic>Universities</topic><topic>Writing Evaluation</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Nikolic, Sasha</creatorcontrib><creatorcontrib>Daniel, Scott</creatorcontrib><creatorcontrib>Haque, Rezwanul</creatorcontrib><creatorcontrib>Belkina, Marina</creatorcontrib><creatorcontrib>Hassan, Ghulam M.</creatorcontrib><creatorcontrib>Grundy, Sarah</creatorcontrib><creatorcontrib>Lyden, Sarah</creatorcontrib><creatorcontrib>Neal, Peter</creatorcontrib><creatorcontrib>Sandison, Caz</creatorcontrib><collection>Taylor &amp; Francis</collection><collection>ERIC</collection><collection>ERIC (Ovid)</collection><collection>ERIC</collection><collection>ERIC</collection><collection>ERIC (Legacy Platform)</collection><collection>ERIC( SilverPlatter )</collection><collection>ERIC</collection><collection>ERIC PlusText (Legacy Platform)</collection><collection>Education Resources Information Center (ERIC)</collection><collection>ERIC</collection><collection>CrossRef</collection><collection>Materials Business File</collection><collection>Mechanical &amp; Transportation Engineering Abstracts</collection><collection>Technology Research Database</collection><collection>Engineering Research Database</collection><collection>Materials Research Database</collection><collection>Civil Engineering Abstracts</collection><jtitle>European journal of engineering education</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Nikolic, Sasha</au><au>Daniel, Scott</au><au>Haque, Rezwanul</au><au>Belkina, Marina</au><au>Hassan, Ghulam M.</au><au>Grundy, Sarah</au><au>Lyden, Sarah</au><au>Neal, Peter</au><au>Sandison, Caz</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><ericid>EJ1390178</ericid><atitle>ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity</atitle><jtitle>European journal of engineering education</jtitle><date>2023-07-04</date><risdate>2023</risdate><volume>48</volume><issue>4</issue><spage>559</spage><epage>614</epage><pages>559-614</pages><issn>0304-3797</issn><eissn>1469-5898</eissn><abstract>ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT's performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.</abstract><cop>Abingdon</cop><pub>Taylor &amp; Francis</pub><doi>10.1080/03043797.2023.2213169</doi><tpages>56</tpages><orcidid>https://orcid.org/0000-0002-5364-6011</orcidid><orcidid>https://orcid.org/0000-0002-7528-9713</orcidid><orcidid>https://orcid.org/0000-0002-5475-9640</orcidid><orcidid>https://orcid.org/0000-0002-8831-5327</orcidid><orcidid>https://orcid.org/0000-0002-8641-4479</orcidid><orcidid>https://orcid.org/0000-0002-3305-9493</orcidid><orcidid>https://orcid.org/0000-0002-6636-8807</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 0304-3797
ispartof European journal of engineering education, 2023-07, Vol.48 (4), p.559-614
issn 0304-3797
1469-5898
language eng
recordid cdi_proquest_journals_2827824019
source Taylor and Francis Science and Technology Collection; ERIC
subjects Artificial Intelligence
Artificial intelligence (AI)
assessment
Benchmarking
Chatbots
ChatGPT
Colleges & universities
Computer Assisted Testing
Engineering Education
Evaluation Methods
Foreign Countries
Generative artificial intelligence
GPT-3
Integrity
Interdisciplinary Approach
Performance Based Assessment
Prompting
Shock waves
Thinking Skills
Undergraduate Students
Universities
Writing Evaluation
title ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-27T10%3A44%3A36IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=ChatGPT%20versus%20engineering%20education%20assessment:%20a%20multidisciplinary%20and%20multi-institutional%20benchmarking%20and%20analysis%20of%20this%20generative%20artificial%20intelligence%20tool%20to%20investigate%20assessment%20integrity&rft.jtitle=European%20journal%20of%20engineering%20education&rft.au=Nikolic,%20Sasha&rft.date=2023-07-04&rft.volume=48&rft.issue=4&rft.spage=559&rft.epage=614&rft.pages=559-614&rft.issn=0304-3797&rft.eissn=1469-5898&rft_id=info:doi/10.1080/03043797.2023.2213169&rft_dat=%3Cproquest_cross%3E2827824019%3C/proquest_cross%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c407t-98ceef95ff74b81b0b55c845608e26fb595352c5bee85c8f6f4eb66e448ad4573%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2827824019&rft_id=info:pmid/&rft_ericid=EJ1390178&rfr_iscdi=true