PERCEIVED MATCH BETWEEN VISUAL PARAMETERS AND AUDITORY CORRELATES: AN EXPERIMENTAL MULTIMEDIA INVESTIGATION

Abstract

This paper investigates the relationship between the auditory and visual components of an audio-visual (A-V) composite. Participants (N=28) were asked to rate the perceived degree of “match” between A-V components in a series of randomly presented composites. Manipulated audio parameters included pitch, loudness, timbre, and duration, while visual parameters included color, vertical location, shape, and size. A-V composites were created by combining all possible pairs of audio and visual stimuli using three different magnitudes of change, resulting in 48 A-V stimuli, i.e., 4 x 4 x 3. Data analysis revealed no statistically significant between-subjects differences as a result of either level of musical training or level of visual training. There was, however, a significant within-subjects difference in responses to the audio-visual pairs. Subject mean response ratings suggest the following primary relationships: pitch with vertical location, loudness with size, and timbre with shape. Duration did not pair as “best match” with any visual parameter. Results also revealed that color is equally matched with both pitch and loudness. The long-term goal of this project is to utilize the results of the music perception study reported as a basis for developing an algorithm to transform musical sound into visual animation, based on perceptually salient properties. 1. BACKGROUND Many multimedia software programs are currently available that provide listeners with a visual analog to musical sound (e.g., Windows Media Player, iTunes, WinAmp, Sonique, etc.). However, the specific manner in which – as well as the basis for – transformation of the sonic component into a visual analog is rarely given much consideration. Though the most frequently used algorithm involves calculation of a fast Fourier transform of the complex audio signal and the mapping of these data onto screen coordinates, email communication with developers who created many of these programs reveals that the selection of visual parameters – color, shape, and motion – altered in association with various aspects of the musical sound – frequency, amplitude, and timbre – tends to be arbitrary. The present investigation sought to identify the degree of match between specific aspects of musical sound and those of animated visual images. The long-term goal of this project is to utilize the results of the music perception study reported herein as a basis for developing an algorithm to transform musical sound into visual animation that is based on perceptually salient properties.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 40 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.