Multimodal Spatiotemporal Representation for Automatic Depression Level Detection

Mingyue Niu · Jianhua Tao · Bin Liu · Jian Huang · Zheng Lian

172 citations0/0 files storedIEEE Transactions on Affective Computing

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Emotion and Mood Recognition Speech and Audio Processing Mental Health via Writing

Abstract

Physiological studies have shown that there are some differences in speech and facial activities between depressive and healthy individuals. Based on this fact, we propose a novel spatio-temporal attention (STA) network and a multimodal attention feature fusion (MAFF) strategy to obtain the multimodal representation of depression cues for predicting the individual depression level. Specifically, we first divide the speech amplitude spectrum/video into fixed-length segments and input these segments into the STA network, which not only integrates the spatial and temporal information through attention mechanism, but also emphasizes the audio/video frames related to depression detection. The audio/video segment-level feature is obtained from the output of the last full connection layer of the STA network. Second, this article employs the eigen evolution pooling method to summarize the changes of each dimension of the audio/video segment-level features to aggregate them into the audio/video level feature. Third, the multimodal representation with modal complementary information is generated using the MAFF and inputs into the support vector regression predictor for estimating depression severity. Experimental results on the AVEC2013 and AVEC2014 depression databases illustrate the effectiveness of our method.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 172 citations exist upstream.

References

2000Measurement Reliability and Validity 2017A review of affective computing: From unimodal analysis to multimodal fusion 2015A review of depression and suicide risk assessment using speech analysis 2013AVEC 2013 2014AVEC 2014 2009OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit 2012Vocal Acoustic Biomarkers of Depression Severity and Treatment Response 2012Paralinguistics in speech and language—State-of-the-art and the challenge

Cited by

2023Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals 2023Spatial–Temporal Attention Network for Depression Recognition from facial videos 2021A hierarchical depression detection model based on vocal and emotional cues 2022Automatic depression recognition by intelligent speech signal processing: A systematic survey 2022Deep Multi-Modal Network Based Automated Depression Severity Estimation

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.