On the role of crossmodal prediction in audiovisual emotion perception

Sarah Jessen · Sonja A. Kotz

48 citations0/0 files storedFrontiers in Human Neuroscience

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Multisensory perception and integration Neuroscience and Music Perception Action Observation and Synchronization

Abstract

Humans rely on multiple sensory modalities to determine the emotional state of others. In fact, such multisensory perception may be one of the mechanisms explaining the ease and efficiency by which others' emotions are recognized. But how and when exactly do the different modalities interact? One aspect in multisensory perception that has received increasing interest in recent years is the concept of cross-modal prediction. In emotion perception, as in most other settings, visual information precedes the auditory information. Thereby, leading in visual information can facilitate subsequent auditory processing. While this mechanism has often been described in audiovisual speech perception, so far it has not been addressed in audiovisual emotion perception. Based on the current state of the art in (a) cross-modal prediction and (b) multisensory emotion perception research, we propose that it is essential to consider the former in order to fully understand the latter. Focusing on electroencephalographic (EEG) and magnetoencephalographic (MEG) studies, we provide a brief overview of the current research in both fields. In discussing these findings, we suggest that emotional visual information may allow more reliable predicting of auditory information compared to non-emotional visual information. In support of this hypothesis, we present a re-analysis of a previous data set that shows an inverse correlation between the N1 EEG response and the duration of visual emotional, but not non-emotional information. If the assumption that emotional content allows more reliable predicting can be corroborated in future studies, cross-modal prediction is a crucial factor in our understanding of multisensory emotion perception.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 48 citations exist upstream.

References

2006Is neocortex essentially multisensory?1999Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study 2000Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex 2005Visual speech speeds up the neural processing of auditory speech 2009The Natural Statistics of Audiovisual Speech 2008Visual Modulation of Neurons in Auditory Cortex 2008Neuronal oscillations and visual amplification of speech 2008Crossmodal binding through neural coherence: implications for multisensory processing

Cited by

2017Overestimation of threat from neutral faces and voices in social anxiety 2014Neural processing of emotion in multimodal settings

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.