SPEECH INTELLIGIBILITY DERIVED FROM ASYNCHRONOUS PROCESSING OF AUDITORY-VISUAL INFORMATION

Abstract

The current study examines the temporal parameters associated with cross-modal integration of auditory-visual information for sentential material (Harvard/IEEE sentences). The speech signal was filtered into 1/3-octave channels, all of which were discarded (in the primary experiment) save for a low-frequency (298-375 Hz) and a high-frequency (4762-6000 Hz) band. The intelligibility of this audio-only signal ranged (depending on the listener) between 9% and 31% (of the key words correct) for nine normal-hearing subjects. Visual-alone presentation of the same material ranged between 1% and 22% intelligibility. When the audio and video signals are combined and presented in perfect synchrony, intelligibility climbs to an average of 63%. The audio and video signals were systematically desynchronized in symmetrical fashion up to a maximum onset asynchrony of 400 ms. When the audio signal leads the video, intelligibility declines appreciably for even the shortest asynchrony of 40 ms, falling to an asymptotic level of performance for asynchronies of ca. 120 ms and longer for most subjects. In contrast, when the video signal leads the audio, intelligibility remains relatively stable for onset asynchronies up to 160-200 ms, and in some instances may actually improve . Hence, there is a marked asymmetry in the integration of audio and visual information that has important implications for sensorybased models of auditory-visual speech processing. 1. INTRODUCTION Speechreading has been shown to improve speech understanding in difficult communication environments such as those associated with background noise, reverberation and hearing loss. The benefit provided by the integration of auditory and visual speech cues relative to audio information alone arises from at least two separate mechanisms. First, independently derived speech information from the eye and the ear are combined early in the decision process so that what is ambiguous within a single modality is rendered much less so in concert with the other [14][21]. Acoustic cues for place of articulation (e.g., [p] vs. [t] vs. [k]) are often weakly represented or entirely absent in the presence of noise or reverberation, while they are robustly encoded in the visual modality. In contrast, voicing cues (e.g., [p] vs. [b]) are robust acoustically, but are difficult to discern in the visual signal. This complementary nature of bimodal speech cues is an important factor underlying the

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 90 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.