Room 1 performed by Steve Gibson with two trackers. © Steve Gibson, 2005. Used with permission.
Live Visuals, Leonardo Electronic Almanac, Volume 19 Issue 3
ISBN: 978-1-906897-22-2
ISSN: 1071-4391
Volume Editors: Lanfranco Aceti, Steve Gibson and Stefan Müller Arisona
Editor: Ozden Sahin
Simulating Synesthesia In Spatially-Based Real-Time Audio-Visual Performance
by Steve Gibson
In this paper I will describe and present examples of my live audio-visual work for 3D spatial environments. These projects use motion-tracking technology to enable users to interact with sound, light and video using their body movements in 3D space. Specific video examples of one past project (Virtual DJ) and one current project (Virtual VJ) will be shown to illustrate how flexible user interaction is enabled through a complex and precise mapping of 3D space to media control. In these projects audience members can interact with sound, light and video in real-time by simply moving around in space with a tracker in hand. Changes in sound can be synchronized with changes in light and/or real-time visual effects (i.e. music volume = light brightness = video opacity). These changes can be dynamically mapped in real-time to allow the user to consolidate the roles of DJ, VJ and light designer in one interface. This interaction model attempts to reproduce the effect of synesthesia, in which certain people experience light or color in response to music.
Full article is available for download as a pdf here.
Volume 19 Issue 3 of Leonardo Electronic Almanac (LEA) is published online as a free PDF but will also be rolled out as Amazon Print on Demand and will be available on iTunes, iPad, Kindle and other e-publishing outlets.