Deep Mul Timodal Learning for Emotion Recognition in Spoken Language

Yue Gu · Shuhong Chen · Ivan Marsic

54 citations0/0 files storedIEEE International Conference on Acoustics Speech and Signal Processing

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech Recognition and Synthesis Sentiment Analysis and Opinion Mining

Abstract

In this paper, we present a novel deep multimodal framework to predict human emotions based on sentence-level spoken language. Our architecture has two distinctive characteristics. First, it extracts the high-level features from both text and audio via a hybrid deep multimodal structure, which considers the spatial information from text, temporal information from audio, and high-level associations from low-level handcrafted features. Second, we fuse all features by using a three-layer deep neural network to learn the correlations across modalities and train the feature extraction and fusion modules together, allowing optimal global fine-tuning of the entire structure. We evaluated the proposed framework on the IEMOCAP dataset. Our result shows promising performance, achieving 60.4% in weighted accuracy for five emotion categories.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 54 citations exist upstream.

References

2008IEMOCAP: interactive emotional dyadic motion capture database 2015High-level feature representation using recurrent neural network for speech emotion recognition 2017Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition 2015An experimental study of speech emotion recognition based on deep convolutional neural networks 2015Speech emotion recognition with acoustic and lexical features 2011Recognizing Affect from Linguistic Information in 3D Continuous Space

Cited by

2019DNN-based Emotion Recognition Based on Bottleneck Acoustic Features and Lexical Features 2021Effect on speech emotion classification of a feature selection approach using a convolutional neural network 2018Human Conversation Analysis Using Attentive Multimodal Networks with Hierarchical Encoder-Decoder

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.