Metric Learning Based Multimodal Audio-visual Emotion Recognition

Esam Ghaleb · Mirela Carmia Popa · Stylianos Asteriadis

59 citations0/0 files storedIEEE Multimedia

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech and Audio Processing Face and Expression Recognition

Abstract

People express their emotions through multiple channels, such as visual and audio ones. Consequently, automatic emotion recognition can be significantly benefited by multimodal learning. Even-though each modality exhibits unique characteristics; multimodal learning takes advantage of the complementary information of diverse modalities when measuring the same instance, resulting in enhanced understanding of emotions. Yet, their dependencies and relations are not fully exploited in audio–video emotion recognition. Furthermore, learning an effective metric through multimodality is a crucial goal for many applications in machine learning. Therefore, in this article, we propose multimodal emotion recognition metric learning (MERML), learned jointly to obtain a discriminative score and a robust representation in a latent-space for both modalities. The learned metric is efficiently used through the radial basis function (RBF) based support vector machine (SVM) kernel. The evaluation of our framework shows a significant performance, improving the state-of-the-art results on the eNTERFACE and CREMA-D datasets.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 59 citations exist upstream.

References

2014CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset 2013Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions 2006The eNTERFACE’05 Audio-Visual Emotion Database 2016AVEC 2016 2019Deep Learning for Human Affect Recognition: Insights and New Developments 2017Deep Spatio-Temporal Features for Multimodal Emotion Recognition 2018Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition

Cited by

2022A Pre-Trained Audio-Visual Transformer for Emotion Recognition 2020Multimodal Attention-Mechanism For Temporal Emotion Recognition 2024Versatile Audio-Visual Learning for Emotion Recognition 2021An End-To-End Emotion Recognition Framework Based on Temporal Aggregation of Multimodal Information 2024Enhancing Emotion Recognition through Federated Learning: A Multimodal Approach with Convolutional Neural Networks 2022Multimodal Affective Computing With Dense Fusion Transformer for Inter- and Intra-Modality Interactions

Imported from OpenAlex (CC0) — nothing here is generated.