Multi-Modal Attention for Speech Emotion Recognition

Zexu Pan · Zhaojie Luo · Jichen Yang · Haizhou Li

95 citations0/0 files storedInterspeech (USB)

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech Recognition and Synthesis Speech and Audio Processing

Abstract

Emotion represents an essential aspect of human speech that is manifested in speech prosody. Speech, visual, and textual cues are complementary in human communication. In this paper, we study a hybrid fusion method, referred to as multi-modal attention network (MMAN) to make use of visual and textual cues in speech emotion recognition. We propose a novel multi-modal attention mechanism, cLSTM-MMA, which facilitates the attention across three modalities and selectively fuse the information. cLSTM-MMA is fused with other uni-modal sub-networks in the late fusion. The experiments show that speech emotion recognition benefits significantly from visual and textual cues, and the proposed cLSTM-MMA alone is as competitive as other fusion methods in terms of accuracy, but with a much more compact network structure. The proposed hybrid network MMAN achieves state-of-the-art performance on IEMOCAP database for emotion recognition.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 95 citations exist upstream.

References

2008IEMOCAP: interactive emotional dyadic motion capture database 2006Solving the Emotion Paradox: Categorization and the Experience of Emotion 2017A review of affective computing: From unimodal analysis to multimodal fusion 2013The INTERSPEECH 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism 2001Sensory modalities are not separate modalities: plasticity and interactions 2018Emotion recognition using deep learning approach from audio–visual emotional big data 2018Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning 2012Ensemble of SVM trees for multimodal emotion recognition

Cited by

2022Key-Sparse Transformer for Multimodal Speech Emotion Recognition 2022Multi-modal speech emotion recognition using self-attention mechanism and multi-scale fusion framework 2022A multimodal emotion recognition model integrating speech, video and MoCAP

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.