FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders

Pengyu Cheng · Weituo Hao · Siyang Yuan · Shijing Si · Lawrence Carin

62 citations0/0 files storedarXiv (Cornell University)

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Ethics and Social Impacts of AI Topic Modeling Hate Speech and Cyberbullying Detection

Abstract

Pretrained text encoders, such as BERT, have been applied increasingly in various natural language processing (NLP) tasks, and have recently demonstrated significant performance gains. However, recent studies have demonstrated the existence of social bias in these pretrained NLP models. Although prior works have made progress on word-level debiasing, improved sentence-level fairness of pretrained encoders still lacks exploration. In this paper, we proposed the first neural debiasing method for a pretrained sentence encoder, which transforms the pretrained encoder outputs into debiased representations via a fair filter (FairFil) network. To learn the FairFil, we introduce a contrastive learning framework that not only minimizes the correlation between filtered embeddings and bias words but also preserves rich semantic information of the original sentences. On real-world datasets, our FairFil effectively reduces the bias degree of pretrained text encoders, while continuously showing desirable performance on downstream tasks. Moreover, our post-hoc method does not require any retraining of the text encoders, further enlarging FairFil's application space.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 62 citations exist upstream.

References

2019HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings 2019MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations 2018Measuring and Mitigating Unintended Bias in Text Classification 2019On Measuring Social Biases in Sentence Encoders 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems 2019Measuring Bias in Contextualized Word Representations 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings

Cited by

2023Unlearning Bias in Language Models by Partitioning Gradients 2023Causal-Debias: Unifying Debiasing in Pretrained Language Models and Fine-tuning via Causal Invariant Learning 2023An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models 2023Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting 2022MABEL: Attenuating Gender Bias using Textual Entailment Data

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.