Large-scale text analysis using generative language models: A case study in discovering public value expressions in AI patents

Sergio Pelaez · Gaurav Verma · Bárbara Ribeiro · Philip Shapira A

36 citations0/0 files storedQuantitative Science Studies

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Computational and Text Analysis Methods Topic Modeling Sentiment Analysis and Opinion Mining

Abstract

Abstract We put forward a novel approach using a generative language model (GPT-4) to produce labels and rationales for large-scale text analysis. The approach is used to discover public value expressions in patents. Using text (5.4 million sentences) for 154,934 US AI patent documents from the United States Patent and Trademark Office (USPTO), we design a semi-automated, human-supervised framework for identifying and labeling public value expressions in these sentences. A GPT-4 prompt is developed that includes definitions, guidelines, examples, and rationales for text classification. We evaluate the labels and rationales produced by GPT-4 using BLEU scores and topic modeling, finding that they are accurate, diverse, and faithful. GPT-4 achieved an advanced recognition of public value expressions from our framework, which it also uses to discover unseen public value expressions. The GPT-produced labels are used to train BERT-based classifiers and predict sentences on the entire database, achieving high F1 scores for the 3-class (0.85) and 2-class classification (0.91) tasks. We discuss the implications of our approach for conducting large-scale text analyses with complex and abstract concepts. With careful framework design and interactive human oversight, we suggest that generative language models can offer significant assistance in producing labels and rationales.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 36 citations exist upstream.

References

2019HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case 2019Content Analysis: An Introduction to Its Methodology 2015What is an emerging technology?2002Public‐Value Failure: When Efficient Markets May Not Do 2011Public Value Mapping and Science Policy Evaluation 2016Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data 2020The application of text mining methods in innovation research: current state, evolution patterns, and development priorities 2004Tech Mining

Cited by

2025Generative AI approach for inventive process visualisation – enhancing human-AI hybrid understanding and comparing of patents 2026Are AI and environmental technology innovations converging?2025A supervised machine learning approach for assessing grant peer review reports 2025Can LLMs Be Analytical Tools? : A Content Analysis Study Using GPT Focusing on Reliability and Validity 2025Introducing a novel AI-based text mining method illustrated through an analysis of German innovation proposals

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.