Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protoc...

Full description

Saved in:
Bibliographic Details
Published in:Journal of intelligence 2023-04, Vol.11 (4), p.76
Main Authors: Byrd, Nick, Joseph, Brianna, Gongora, Gabriela, Sirota, Miroslav
Format: Article
Language:English
Subjects:
Analysis
Cellular telephone equipment industry
Cognition & reasoning
cognitive reflection test
Data collection
Frederick, Shane
heuristics and biases
Hypotheses
judgment- and decision-making
psychometrics
think-aloud protocol analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523
cites cdi_FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523
container_end_page
container_issue 4
container_start_page 76
container_title Journal of intelligence
container_volume 11
creator Byrd, Nick
Joseph, Brianna
Gongora, Gabriela
Sirota, Miroslav
description The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).
doi_str_mv 10.3390/jintelligence11040076
format article
fullrecord <record><control><sourceid>gale_doaj_</sourceid><recordid>TN_cdi_doaj_primary_oai_doaj_org_article_b219883459f74b6b882366f321aa3e9e</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><galeid>A751984448</galeid><doaj_id>oai_doaj_org_article_b219883459f74b6b882366f321aa3e9e</doaj_id><sourcerecordid>A751984448</sourcerecordid><originalsourceid>FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523</originalsourceid><addsrcrecordid>eNptkl1rFDEUhgdRbKn9CUrAG2-25nOSeCNL8aNQUGSreBUymZPdrNlJm8wU9t-bdWppxeQih5P3fZJzOE3zkuAzxjR-uw3DCDGGNQwOCMEcY9k-aY4plnrBKMZPH8RHzWkpW1yXJkxx-rw5YpJgRlty3PhV5aCrgn5s7Ih-pgl9AxvjHq02Yfj1Di3nAC1jmnr0NacxuRTRcrBxX0JByaNxA-g75M5GdJ7WQxjDLVSKj-DGkAa0gjK-aJ55Gwuc3p0nzdXHD6vzz4vLL58uzpeXCye4GBcStHNOKwBBRQut9dB3wKnizBPq215Lqpn0hFDhoGMd6YkUijgvmaKCspPmYub2yW7NdQ47m_cm2WD-JFJeG5vH4CKYjhKtFONCe8m7tlOKsrb1jBJrGWiorPcz63rqdtA7GMZs4yPo45shbMw63RqCCW-F1pXw5o6Q081U22B2objacDtAmoqhCrdVJsnh46__kW7TlGuXZ5Xgte6D6mxWrW2tIAw-1Ydd3T3sgksD-FDzSylqaZxzVQ1iNricSsng779PsDmMkvnvKFXfq4e137v-Dg77DXhCxsA</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2806542932</pqid></control><display><type>article</type><title>Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test</title><source>Publicly Available Content Database</source><source>PubMed Central(OpenAccess)</source><source>Coronavirus Research Database</source><creator>Byrd, Nick ; Joseph, Brianna ; Gongora, Gabriela ; Sirota, Miroslav</creator><creatorcontrib>Byrd, Nick ; Joseph, Brianna ; Gongora, Gabriela ; Sirota, Miroslav</creatorcontrib><description>The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).</description><identifier>ISSN: 2079-3200</identifier><identifier>EISSN: 2079-3200</identifier><identifier>DOI: 10.3390/jintelligence11040076</identifier><identifier>PMID: 37103261</identifier><language>eng</language><publisher>Switzerland: MDPI AG</publisher><subject>Analysis ; Cellular telephone equipment industry ; Cognition &amp; reasoning ; cognitive reflection test ; Data collection ; Frederick, Shane ; heuristics and biases ; Hypotheses ; judgment- and decision-making ; psychometrics ; think-aloud protocol analysis</subject><ispartof>Journal of intelligence, 2023-04, Vol.11 (4), p.76</ispartof><rights>COPYRIGHT 2023 MDPI AG</rights><rights>2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>2023 by the authors. 2023</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523</citedby><cites>FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523</cites><orcidid>0000-0001-5475-5941</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.proquest.com/docview/2806542932/fulltextPDF?pq-origsite=primo$$EPDF$$P50$$Gproquest$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://www.proquest.com/docview/2806542932?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>230,314,727,780,784,885,25753,27924,27925,37012,37013,38516,43895,44590,53791,53793,74412,75126</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/37103261$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Byrd, Nick</creatorcontrib><creatorcontrib>Joseph, Brianna</creatorcontrib><creatorcontrib>Gongora, Gabriela</creatorcontrib><creatorcontrib>Sirota, Miroslav</creatorcontrib><title>Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test</title><title>Journal of intelligence</title><addtitle>J Intell</addtitle><description>The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).</description><subject>Analysis</subject><subject>Cellular telephone equipment industry</subject><subject>Cognition &amp; reasoning</subject><subject>cognitive reflection test</subject><subject>Data collection</subject><subject>Frederick, Shane</subject><subject>heuristics and biases</subject><subject>Hypotheses</subject><subject>judgment- and decision-making</subject><subject>psychometrics</subject><subject>think-aloud protocol analysis</subject><issn>2079-3200</issn><issn>2079-3200</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>COVID</sourceid><sourceid>PIMPY</sourceid><sourceid>DOA</sourceid><recordid>eNptkl1rFDEUhgdRbKn9CUrAG2-25nOSeCNL8aNQUGSreBUymZPdrNlJm8wU9t-bdWppxeQih5P3fZJzOE3zkuAzxjR-uw3DCDGGNQwOCMEcY9k-aY4plnrBKMZPH8RHzWkpW1yXJkxx-rw5YpJgRlty3PhV5aCrgn5s7Ih-pgl9AxvjHq02Yfj1Di3nAC1jmnr0NacxuRTRcrBxX0JByaNxA-g75M5GdJ7WQxjDLVSKj-DGkAa0gjK-aJ55Gwuc3p0nzdXHD6vzz4vLL58uzpeXCye4GBcStHNOKwBBRQut9dB3wKnizBPq215Lqpn0hFDhoGMd6YkUijgvmaKCspPmYub2yW7NdQ47m_cm2WD-JFJeG5vH4CKYjhKtFONCe8m7tlOKsrb1jBJrGWiorPcz63rqdtA7GMZs4yPo45shbMw63RqCCW-F1pXw5o6Q081U22B2objacDtAmoqhCrdVJsnh46__kW7TlGuXZ5Xgte6D6mxWrW2tIAw-1Ydd3T3sgksD-FDzSylqaZxzVQ1iNricSsng779PsDmMkvnvKFXfq4e137v-Dg77DXhCxsA</recordid><startdate>20230421</startdate><enddate>20230421</enddate><creator>Byrd, Nick</creator><creator>Joseph, Brianna</creator><creator>Gongora, Gabriela</creator><creator>Sirota, Miroslav</creator><general>MDPI AG</general><general>MDPI</general><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>4U-</scope><scope>7XB</scope><scope>88I</scope><scope>8FK</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>CCPQU</scope><scope>COVID</scope><scope>DWQXO</scope><scope>GNUQQ</scope><scope>HCIFZ</scope><scope>M2P</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PSYQQ</scope><scope>Q9U</scope><scope>7X8</scope><scope>5PM</scope><scope>DOA</scope><orcidid>https://orcid.org/0000-0001-5475-5941</orcidid></search><sort><creationdate>20230421</creationdate><title>Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test</title><author>Byrd, Nick ; Joseph, Brianna ; Gongora, Gabriela ; Sirota, Miroslav</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Analysis</topic><topic>Cellular telephone equipment industry</topic><topic>Cognition &amp; reasoning</topic><topic>cognitive reflection test</topic><topic>Data collection</topic><topic>Frederick, Shane</topic><topic>heuristics and biases</topic><topic>Hypotheses</topic><topic>judgment- and decision-making</topic><topic>psychometrics</topic><topic>think-aloud protocol analysis</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Byrd, Nick</creatorcontrib><creatorcontrib>Joseph, Brianna</creatorcontrib><creatorcontrib>Gongora, Gabriela</creatorcontrib><creatorcontrib>Sirota, Miroslav</creatorcontrib><collection>PubMed</collection><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>University Readers</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Science Database (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>ProQuest One Community College</collection><collection>Coronavirus Research Database</collection><collection>ProQuest Central Korea</collection><collection>ProQuest Central Student</collection><collection>SciTech Premium Collection</collection><collection>Science Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>ProQuest One Psychology</collection><collection>ProQuest Central Basic</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><collection>DOAJ Directory of Open Access Journals</collection><jtitle>Journal of intelligence</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Byrd, Nick</au><au>Joseph, Brianna</au><au>Gongora, Gabriela</au><au>Sirota, Miroslav</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test</atitle><jtitle>Journal of intelligence</jtitle><addtitle>J Intell</addtitle><date>2023-04-21</date><risdate>2023</risdate><volume>11</volume><issue>4</issue><spage>76</spage><pages>76-</pages><issn>2079-3200</issn><eissn>2079-3200</eissn><abstract>The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).</abstract><cop>Switzerland</cop><pub>MDPI AG</pub><pmid>37103261</pmid><doi>10.3390/jintelligence11040076</doi><orcidid>https://orcid.org/0000-0001-5475-5941</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 2079-3200
ispartof Journal of intelligence, 2023-04, Vol.11 (4), p.76
issn 2079-3200
2079-3200
language eng
recordid cdi_doaj_primary_oai_doaj_org_article_b219883459f74b6b882366f321aa3e9e
source Publicly Available Content Database; PubMed Central(OpenAccess); Coronavirus Research Database
subjects Analysis
Cellular telephone equipment industry
Cognition & reasoning
cognitive reflection test
Data collection
Frederick, Shane
heuristics and biases
Hypotheses
judgment- and decision-making
psychometrics
think-aloud protocol analysis
title Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-28T18%3A19%3A47IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-gale_doaj_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Tell%20Us%20What%20You%20Really%20Think:%20A%20Think%20Aloud%20Protocol%20Analysis%20of%20the%20Verbal%20Cognitive%20Reflection%20Test&rft.jtitle=Journal%20of%20intelligence&rft.au=Byrd,%20Nick&rft.date=2023-04-21&rft.volume=11&rft.issue=4&rft.spage=76&rft.pages=76-&rft.issn=2079-3200&rft.eissn=2079-3200&rft_id=info:doi/10.3390/jintelligence11040076&rft_dat=%3Cgale_doaj_%3EA751984448%3C/gale_doaj_%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c545t-7e9ccc98ee5256e6afedbe42843f12f6d972937f1125ceb3b1d17581cf7382523%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2806542932&rft_id=info:pmid/37103261&rft_galeid=A751984448&rfr_iscdi=true