A Pre-Trained Audio-Visual Transformer for Emotion Recognition

Minh Tam Tran · Mohammad Soleymani

44 citations0/0 files storedIEEE International Conference on Acoustics Speech and Signal Processing

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech Recognition and Synthesis Speech and Audio Processing

Abstract

In this paper, we introduce a pretrained audio-visual Transformer trained on more than 500k utterances from nearly 4000 celebrities from the VoxCeleb2 dataset for human behavior understanding. The model aims to capture and extract useful information from the interactions between human facial and auditory behaviors, with application in emotion recognition. We evaluate the model performance on two datasets, namely CREMAD-D (emotion classification) and MSP-IMPROV (continuous emotion regression). Experimental results show that fine-tuning the pre-trained model helps improving emotion classification accuracy by 5-7% and Concordance Correlation Coefficients (CCC) in continuous emotion recognition by 0.03-0.09 compared to the same model trained from scratch. We also demonstrate the robustness of finetuning the pre-trained model in a low-resource setting. With only 10% of the original training set provided, finetuning the pre-trained model can lead to at least 10% better emotion recognition accuracy and a CCC score improvement by at least 0.1 for continuous emotion recognition.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 44 citations exist upstream.

References

2014CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset 2016MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception 2017Jointly Predicting Arousal, Valence and Dominance with Multi-Task Learning 2019Metric Learning Based Multimodal Audio-visual Emotion Recognition

Cited by

2024Versatile Audio-Visual Learning for Emotion Recognition 2024Joint Multimodal Transformer for Emotion Recognition in the Wild 2025Joint Multi-Scale Multimodal Transformer for Emotion Using Consumer Devices 2022Recognition of Emotions in Speech Using Convolutional Neural Networks on Different Datasets

Imported from OpenAlex (CC0) — nothing here is generated.