A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations

Abstract

Multimodal Emotion Recognition in Multiparty Conversations (MERMC) has recently attracted considerable attention.Due to the complexity of visual scenes in multi-party conversations, most previous MERMC studies mainly focus on text and audio modalities while ignoring visual information.Recently, several works proposed to extract face sequences as visual features and have shown the importance of visual information in MERMC.However, given an utterance, the face sequence extracted by previous methods may contain multiple people's faces, which will inevitably introduce noise to the emotion prediction of the real speaker.To tackle this issue, we propose a two-stage framework named Facial expressionaware Multimodal Multi-Task learning (Fa-cialMMT).Specifically, a pipeline method is first designed to extract the face sequence of the real speaker of each utterance, which consists of multimodal face recognition, unsupervised face clustering, and face matching.With the extracted face sequences, we propose a multimodal facial expression-aware emotion recognition model, which leverages the frame-level facial emotion distributions to help improve utterance-level emotion recognition based on multi-task learning.Experiments demonstrate the effectiveness of the proposed FacialMMT framework on the benchmark MELD dataset.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 51 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.