Goodbye human annotators? Content analysis of social policy debates using ChatGPT

Erwin Gielens · Jakub Sowula · Philip Leifeld

13 citations0/0 files storedJournal of Social Policy

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Computational and Text Analysis Methods Sentiment Analysis and Opinion Mining Artificial Intelligence in Law

Abstract

Abstract Content analysis is a valuable tool for analysing policy discourse, but annotation by humans is costly and time consuming. ChatGPT is a potentially valuable tool to partially automate content analysis for policy debates, largely replacing human annotators. We evaluate ChatGPT’s ability to classify documents using pre-defined argument descriptions, comparing its performance with human annotators for two policy debates: the Universal Basic Income debate on Dutch Twitter (2014–2016) and the pension reforms debate in German newspapers (1993–2001). We use the API (GPT-4 Turbo) and user interface version (GPT-4) and evaluate multiple performance metrics (accuracy, precision and recall). ChatGPT is highly reliable and accurate in classifying pre-defined arguments across datasets. However, precision and recall are much lower, and vary strongly between arguments. These results hold for both datasets, despite differences in language and media type. Moreover, the cut-off method proposed in this paper may aid researchers in navigating the trade-off between detection and noise. Overall, we do not (yet) recommend a blind application of ChatGPT to classify arguments in policy debates. Those interested in adopting this tool should manually validate bot classifications before using them in further analyses. At least for now, human annotators are here to stay.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 13 citations exist upstream.

References

2005Three Approaches to Qualitative Content Analysis 2008Discursive Institutionalism: The Explanatory Power of Ideas and Discourse 2013Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts 2023ChatGPT outperforms crowd workers for text-annotation tasks 2008A Scaling Model for Estimating Time‐Series Party Positions from Texts 2023Can Large Language Models Transform Computational Social Science?2023Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT 2013Reconceptualizing Major Policy Change in the Advocacy Coalition Framework: A Discourse Network Analysis of German Pension Politics

Cited by

2025Content moderation by LLM: from accuracy to legitimacy 2025ChatGPT’s Potential for Quantitative Content Analysis: Categorizing Actors in German News Articles 2026Re‐Imagining the Epistemic Possibilities of GPT for Public Administration Research in Competitive Settings

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.