Multimodal Emotion Recognition with High-Level Speech and Text Features

Mariana Rodrigues Makiuchi · Kuniaki Uto · Koichi Shinoda

89 citations0/0 files storedIEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech Recognition and Synthesis Sentiment Analysis and Opinion Mining

Abstract

Automatic emotion recognition is one of the central concerns of the Human-Computer Interaction field as it can bridge the gap between humans and machines. Current works train deep learning models on low-level data representations to solve the emotion recognition task. Since emotion datasets often have a limited amount of data, these approaches may suffer from overfitting, and they may learn based on superficial cues. To address these issues, we propose a novel cross-representation speech model, inspired by disentangle-ment representation learning, to perform emotion recognition on wav2vec 2.0 speech features. We also train a CNN-based model to recognize emotions from text features extracted with Transformer-based models. We further combine the speech-based and text-based results with a score fusion approach. Our method is evaluated on the IEMOCAP dataset in a 4-class classification problem, and it surpasses current works on speech-only, text-only, and multimodal emotion recognition.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 89 citations exist upstream.

References

2008IEMOCAP: interactive emotional dyadic motion capture database 2010Survey on speech emotion recognition: Features, classification schemes, and databases 2003Speech emotion recognition using hidden Markov models 2014Speech emotion recognition using deep neural network and extreme learning machine 2021Emotion Recognition from Speech Using wav2vec 2.0 Embeddings 2005Speech emotion recognition based on HMM and SVM 2012Speech Emotion Recognition Using Support Vector Machine 2020Jointly Fine-Tuning “BERT-Like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Cited by

2024A systematic review of trimodal affective computing approaches: Text, audio, and visual integration in emotion recognition and sentiment analysis 2022A novel dual-modal emotion recognition algorithm with fusing hybrid features of audio signal and speech context 2022Multimodal Emotion Recognition Using Transfer Learning from Speaker Recognition and BERT-Based Models 2023Deep Learning Approaches for Bimodal Speech Emotion Recognition: Advancements, Challenges, and a Multi-Learning Model

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.