Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations

Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitat...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-11
Main Authors: Yan, Lixiang, Echeverria, Vanessa, Gloria Fernandez Nieto, Jin, Yueqiao, Zachari Swiecki, Zhao, Linxuan, Gašević, Dragan, Martinez-Maldonado, Roberto
Format: Article
Language:English
Subjects:
Chatbots
Collaboration
Cooperation
Feedback loops
Generative artificial intelligence
Machine learning
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.
ISSN:2331-8422