Debiasing Pre-Trained Language Models via Efficient Fine-Tuning

Michael Gira · Ruisu Zhang · Kangwook Lee

32 citations0/0 files storedWorkshop on Language Technology for Equality, Diversity and Inclusion (LT-EDI)

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Ethics and Social Impacts of AI Topic Modeling Hate Speech and Cyberbullying Detection

Abstract

An explosion in the popularity of transformerbased language models (such as GPT-3, BERT, RoBERTa, and ALBERT) has opened the doors to new machine learning applications involving language modeling, text generation, and more.However, recent scrutiny reveals that these language models contain inherent biases towards certain demographics reflected in their training data.While research has tried mitigating this problem, existing approaches either fail to remove the bias completely, degrade performance ("catastrophic forgetting"), or are costly to execute.This work examines how to reduce gender bias in a GPT-2 language model by fine-tuning less than 1% of its parameters.Through quantitative benchmarks, we show that this is a viable way to reduce prejudice in pre-trained language models while remaining cost-effective at scale.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 32 citations exist upstream.

References

2019HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case 2023Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics 1998Measuring individual differences in implicit cognition: The implicit association test.2021A Survey on Bias and Fairness in Machine Learning 2017Semantics derived automatically from language corpora contain human-like biases 2015Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition 2019Actionable Auditing 2021StereoSet: Measuring stereotypical bias in pretrained language models

Cited by

2023An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models 2023Improving Gender-Related Fairness in Sentence Encoders: A Semantics-Based Approach 2024Debiasing large language models: research opportunities*

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.