Multimodal Transformer Fusion for Continuous Emotion Recognition

Jian Huang · Jianhua Tao · Bin Liu · Zheng Lian · Mingyue Niu

187 citations0/0 files storedIEEE International Conference on Acoustics Speech and Signal Processing

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech and Audio Processing Sentiment Analysis and Opinion Mining

Abstract

Multimodal fusion increases the performance of emotion recognition because of the complementarity of different modalities. Compared with decision level and feature level fusion, model level fusion makes better use of the advantages of deep neural networks. In this work, we utilize the Transformer model to fuse audio-visual modalities on the model level. Specifically, the multi-head attention produces multimodal emotional intermediate representations from common semantic feature space after encoding audio and visual modalities. Meanwhile, it also can learn long-term temporal dependencies with self-attention mechanism effectively. The experiments, on the AVEC 2017 database, shows the superiority of model level fusion than other fusion strategies. Moreover, we combine the Transformer model and LSTM to further improve the performance, which achieves better results than other methods.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 187 citations exist upstream.

References

1989A Concordance Correlation Coefficient to Evaluate Reproducibility 2015The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing 2004Analysis of emotion recognition using facial expressions, speech and multimodal information 2016AVEC 2016 2010Automatic, Dimensional and Continuous Emotion Recognition 2013Deep learning for robust feature generation in audiovisual emotion recognition 2017AVEC 2017 2014Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild

Cited by

2023A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods 2020Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition 2024Using transformers for multimodal emotion recognition: Taxonomies and state of the art review 2025A Comprehensive Review of Multimodal Emotion Recognition: Techniques, Challenges, and Future Directions 2021Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.