Articlegold2025

Leveraging ChatGPT for thematic analysis of medical best practice advisory data

Yejin Jeong · Margaret A. Smith · Robert J. Gallo · Lisa Marie Knowlton · Steven Lin · Lisa Y. Shieh

7 citations0/0 files storedJAMIA Open

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Computational and Text Analysis Methods Artificial Intelligence in Healthcare and Education Qualitative Research Methods and Applications

Abstract

Objectives: To evaluate ChatGPT's ability to perform thematic analysis of medical Best Practice Advisory (BPA) free-text comments and identify prompt engineering strategies that optimize performance. Materials and Methods: We analyzed 778 BPA comments from a pilot AI-enabled clinical deterioration intervention at Stanford Hospital, categorized as reasons for deterioration (Category 1) and care team actions (Category 2). Prompt engineering strategies (role, context specification, stepwise instructions, few-shot prompting, and dialogue-based calibration) were tested on a 20% random subsample to determine the best-performing prompt. Using that prompt, ChatGPT conducted deductive coding on the full dataset followed by inductive analysis. Agreement with human coding was assessed as inter-rater reliability (IRR) using Cohen's Kappa (κ). Results: With structured prompts and calibration, ChatGPT achieved substantial agreement with human coding (κ = 0.76 for Category 1; κ = 0.78 for Category 2). Baseline agreement was higher for Category 1 than Category 2, reflecting differences in comment type and complexity, but calibration improved both. Inductive analysis yielded 9 themes, with ChatGPT-generated themes closely aligning with human coding. Discussion: ChatGPT can accelerate qualitative analysis, but its rigor depends heavily on prompt engineering. Key strategies included role and context specification, pulse-check calibration, and safeguard techniques, which enhanced reliability and reproducibility. Conclusion: This study demonstrates the feasibility of ChatGPT-assisted thematic analysis and introduces a structured approach for applying LLMs to qualitative analysis of clinical free-text data, underscoring prompt engineering as a methodological lever.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 7 citations exist upstream.

References

2023Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT 2024An Examination of the Use of Large Language Models to Aid Analysis of Textual Data 2023Artificial Intelligence Augmented Qualitative Analysis: The Way of the Future?2024Harnessing ChatGPT for Thematic Analysis: Are We Ready?2024Comparing the Efficacy and Efficiency of Human and Generative AI: Qualitative Thematic Analyses 2025Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance 2024The Value of Generative AI for Qualitative Research: A Pilot Study 2024Doing Research with Help from ChatGPT: Promising Examples for Coding and Inter-Rater Reliability

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.