Performing Sonic City: Situated Creativity In Mobile Music Making

by Lalya Gaye and Lars Erik Holmquist
Future Applications Lab
Viktoria Institute
Hörselgången 4, 417 56 Göteborg
Sweden
lalya [@] viktoria [dot] se
leh [@] viktoria [dot] se
http://www.viktoria.se/fal/projects/soniccity/

KEYWORDS
mobile media, ubiquitous computing, interactive music, mobile music making, everyday creativity, situated interaction.

ABSTRACT
*Sonic City* is a mobile music application that turns the city into a musical interface. A study with participants using our prototype in their everyday settings showed how *Sonic City* mediates a new type of personal experience of urban space and embeds electronic music making in the everyday. This paper sets *Sonic City* and the notion of mobile music making into perspective, describes how the characteristics of this type of systems affect user behaviour and experience, and discusses implications for this emerging field.

_____________________________

1. INTRODUCTION
*Sonic City* is a wearable system that turns the city into an interface for real time electronic music making. It enables its user to create a personal soundscape of live electronic music by walking through and interacting with urban environments. The prototype consists of a small laptop computer, a microphone, headphones, a micro-controller, a MIDI interface, and a number of sensors (sensing light, metal, movement, proximity, sound level, etc). The system gathers information about the user’s actions and surrounding context with sensors worn on the body and a layer of context and action recognition. This data controls the audio processing of live urban sounds collected by the microphone. Resulting music is output through headphones in real time and in the context in which it is created, as the user is walking. Mobility through shifting urban context becomes a large-scale musical gesture. (For more details, see [8]).

*Sonic City* is one example of *mobile music making*. This paper sets this project and the notion of mobile music making into perspective, describes how the characteristics of this type of systems affect user behaviour and experience, and discusses implications for this emerging field.

2. MOBILITY AND AESTHETIC CREATIVITY IN THE EVERYDAY
When new technologies emerge and infiltrate society, they often enable new forms of aesthetic practices for artists and everyday people. Ubiquitous and mobile technologies situate computing into the real world with features such as context and location sensitivity, merging the physical and the digital realms. This gives new dimensions to everyday activities and settings, and invests them with new social meaning and aesthetic values. Thus, mobile computing can give rise to new types of creative behaviours and aesthetic practices in people's everyday life and allows new types of media - mobile media - to develop.

2.1 Aesthetic Mobile Practices

Although the field of mobile media is relatively new, practices embedded in the everyday where mobility becomes a creative act have existed for a long time.

Walking is the most ancient form of aesthetic mobile practice. Because of the relation it creates between body and space, the art of walking can become an act of introspection, a critique of public space, a political act, or an aesthetic practice [3]. In Aboriginal walkabouts, a system of routes mapping the whole Australian continent is connected to song-lines and tales of the origins of mankind; walking reinterprets these narratives, bringing them to life. In the situationist dérive - an aimless, explorative and playful drifting through city, or "technique of rapid passage through varied ambiences" [4] - walking becomes a means of shaking one's perception of everyday urban space and creating new meaning within it: one walks to get lost in the city and break from the monotony of usual everyday paths, exploring unknown urban areas and one's awareness of own mental maps.

In other mobile practices, such as the urban sports of parkour or skateboarding, mobility turns the urban environment into a physical resource for aesthetic performance. In street skateboarding, people repurpose and interface with urban space through their decks, giving new meanings to pavements and architecture through their physical use [2]. In the discipline of parkour [15], an extreme French artistic urban sport making use of architectural infrastructures and urban furnish, practitioners ('traceurs') use urban space in a creative way by climbing buildings with their bare hands, jumping over fences or across staircases. Traceurs and skateboarders transform what most of us would consider obstacles or barriers (such as walls or fences) into resources for movement, attaching great importance to mobile aesthetics: while moving through the city or exploring one particular spot, they approach urban space in terms of how it can be used, what resources are available at hand for jumps, climbing, etc.

2.2 Mobile Music Making

Mapped to the physical world with ubiquitous and mobile technology, mobile media differs from other types of new media by its strong connection to place and its active awareness of location or context. It resituates media contents and interaction into the real world and thus into the everyday, encompassing fields such as augmented (e.g. [7]) and mixed-reality (e.g. [1]). Interactive music is another exciting field where users also break out from desktop settings and the 'computer screen - mouse' paradigm, and that puts the focus of electronic music back into real-time music making and physical interaction in live performances.

Mobile music making can be considered to be at the intersection of these two fields. In its own turn, this new field breaks out of the desktop and music performance settings, and situates the act of making electronic music into the real everyday world. Enabling to making music on the go, mobile music devices can make use of their physical context (e.g. *Sonic City*), social context (e.g. [18]) or be portable devices without sensing capabilities that are easily used in mobile settings (e.g. [16]). Various projects have been dilating the space of possibilities that everyday people have grown accustomed to since the Sony walkman: already in 1969, Wodiczko created the *Personal Instrument* [19], a wearable that filters sound in headphones based on the intensity of light hitting the user's palms; more recently, *Sound Lens* [10] is a portable device converting flickering of artificial light into sound. Other projects enable collaborative mobile music making in the city: in *Sound Mapping* [13] - a site-specific outdoor interactive music event, co-located people interacted and created music with a set of portable suitcases equipped with sensors and GPS; in *CosTune* [14] users are connected by a network of wearables that can be explicitly played with touch; in *Malleable Mobile Music* [18] - a collaborative proximity and location-based music remixer, each user contributes with a track to the music. Local soundscapes can also be reinterpreted on the move: *Noiseman* [6] and *Sonic Interface* [11] filter and remix urban sounds with pre-programmed audio units carried in backpacks, while *Nomadic Audio by Mazé* [12] retrieves local radio frequencies for individual train commuters. Locative soundscape projects such as *Trace* [17] connect music to location with the help of GPS, in a way reminiscent of the aboriginal walkabouts.

3. *SONIC CITY*: MOBILE MUSICAL INTERACTION WITH URBAN SPACE
In the mobile music application *Sonic City*, mobility itself becomes the main interaction vector: the user's movement through shifting urban context and her interaction with the physical environment creates new music in real time. By relying on everyday mobile settings and actions, and creating a link between the constructed space of the city and the constructed time of music, *Sonic City* mediates a new type of relationship between the user and her surrounding physical urban space that embeds electronic music making in the everyday. The following characteristics of *Sonic City* imply interesting interaction properties that can hint on the future potentials of mobile music making. The music produced with *Sonic City* is created based on the surrounding context, as well as experienced in context, as it is being created. The user only interacts with her local context, not with the city as a whole. This brings a dimension of immediacy to the interaction and makes the musical experience situated. Because the system is wearable, the space of enabled interactions is also user-centric and non site-specific. This interaction space is however scaled to the dimensions of a city: the musical time-line is matched to the user's path and the time it takes to travel certain distances. This implies a certain scale of musical gesture. In terms of musical gesture, mobile interactions relying on everyday, normally non-musical embodied actions such as walking are devoid of a specific gestural language for music. In terms of musical interface, the city itself was obviously not either designed for the specific purpose of making music. However, *Sonic City* projects a space of designed interactions on the urban space: the city is repurposed into a musical interface. *Sonic City* provides entry points to apprehending the city's everyday settings, objects and actions at hand as music makers, and to repurposing them as resources for musical interaction, resources to physically *use* and interact *with*. In the city - a constructed and familiar, yet dynamic and unpredictable interface - the user's immediate surroundings shift in time and along her path: the interface is in constant transition, dynamic and heterogeneous.

4. MOBILE MUSIC MAKING WITH SONIC CITY
What do the characteristics of *Sonic City* imply for the notion of mobility as a musical act, in terms of user experience and behavior? To explore this question, we conducted a user study with five participants using our prototype in their own everyday environments, within the city of Göteborg. (For more information about the user study, see [9]).

4.1 User Experience and Behaviors

The user experience alternated among different states: immersive rediscovery of everyday urban space; background music listening while managing other mobile factors such as traffic and social contexts; and active engagement into the music creation. Using *Sonic City* enhanced the users' perception of and engagement with their everyday settings. They felt more aware of details highlighted by the system, of things that they had stopped paying attention to or never even noticed. However, they also perceived that the city was more in control of the music than they were, due to unpredictable and uncontrollable factors encountered in urban environments that had more effect on the music than their own actions. This pushed the users to actively regain control over the music, which they would do through various ad hoc tactics, both on a path level and on a local immediate level. The users modified their planned paths in order to search for unusual urban contexts (electrical chamber, etc), and engaged in local interactions with shifting resources at hand, directing sensors with their body towards sources of input (such as metal) or modulated the city's input by shadowing sensors from noise or light with their body posture. Paths could be seen as scores articulated by ad hoc local bodily interactions.

4.2 Augmented Walks: Interweaving Activities of Walking and Music Making

The experience alternated between being active and passive, going back and forth from an immersive experience to background music listening, to active interventions in the music. During active phases, users looked for sources of input and interacted with them. The experience would become passive when the activity of dealing with the city had higher priority or when the users wanted to simply hear what the city did musically, at which point the experience would become more introspective and intimate. Navigation through space alternated as well between being motivated by intentional musical actions (such as suddenly getting closer to a wall) and by normal everyday mobile behaviours (crossing a street, avoiding a dog...) Sometimes, users even made musical actions pass as everyday activity, pretending for example to be looking at a shop window when actually aiming to hide the microphone from loud traffic noises. Mobile music making in real world settings can therefore alternate between a foreground and background activity, something that should be supported in the design of mobile music applications.

4.3 Situated Creative Behaviours

Due to the shifting and repurposed nature of the city as musical interface, users wishing to regain control over the experience had to make the effort of adapting their interactions in situations, to dynamic, heterogeneous and sometimes unpredictable resources at hand: they looked for interesting context and local interactions to engage into in a rather ad hoc way and often found some by accident. The resulting music was a reflection of these situated actions and was highly contextual: the more dynamic and complex the context, the richer the music. This type of situated improvisation in mobile music making can be related to the practice of 'bricolage', a form of everyday creativity and inventiveness: making do with what is at hand in a particular situation, either to find a solution to a problem, or as in our case to create something new with available materials [5]. Therefore, even when implying a loss of control from the user's part, heterogeneity and unpredictability are interesting aspects of urban interactions to embrace in the design of mobile music applications, as they enrich the experience during passive phases and stimulate user participation during active ones.

5. CONCLUSIONS
At the intersection of mobile media and interactive music, the aesthetic practice of mobile music making mediates a new type of personal experience of urban space while embedding electronic music making in the real everyday world. *Sonic City* in action showed that mobility could become a new and original type of technology-mediated experience that alters between rediscovery of the everyday and creativity in situation with shifting physical everyday resources at hand. With the current explosion of mobile technology, could such new types of practices become widespread in the near future?

ILLUSTRATIONS

Figure 1. Mobile music making with Sonic City: a user in action

Figure 2. Sonic City user study participants

All images copyrighted © Gaye and Holmquist

ACKNOWLEDGMENTS
*Sonic City* is a collaboration between Future Applications Lab, Viktoria Institute, and Re:Form studio, Interactive Institute, funded by the Swedish Foundation for Strategic Research through the Mobile Services project, and by the European IST project ECAgents (IST-1940). Thanks to the study participants, to Anne Galloway, Chris Salter, Barry Brown, and project partners Ramia Mazé, Daniel Skoglund, and Margot Jacobs for inspiration and valuable comments and discussions.
_____________________________

REFERENCES

1. M. Adams, J. Row Farr, N. Tandavanitj, S. Benford, M. Flintham, A. Drozd, R. Anastasi, "Can You See Me Now?", *Video proc. UbiComp'04*.

2. I. Borden, *Skateboarding, Space and the City: Architecture and the Body* (Berg Publishers, 2001).

3. F. Careri, *Walkscapes: Walking as an Aesthetic Practice* (Land&Scape Series, 2002).

4. G.-E. Debord, "Theory of the Dérive", in *Les Lèvres Nues #9* (1956).

5. M. de Certeau, *The Practice of Everyday Life* (1984).

6. A. Dunne, *Noiseman*, http://www.dunneandraby.co.uk/

7. S. Feiner, B. MacIntyre, T. Hollerer, and T. Webster, "A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment", in *Personal Technology*, Vol. 1, No.4 (1997).

8. L. Gaye, R. Mazé, L.E. Holmquist, "Sonic City: The Urban Environment as a Musical Interface", *Proc. NIME'03*.

9. Gaye, L., Holmquist, L. E., "In Duet with Everyday Urban Settings: A User Study of Sonic City", *Proc. NIME'04*.

10. T. Iwai, *Sound Lens*, http://www.2dk.net/movie/e_index.html

11. A. Maebayashi, *Sonic Interface*, http://www2.gol.com/users/m8/msonic.html

12. R. Mazé, *Nomadic Audio*, http://www.tii.se/reform/people/maze_ramia.html

13. I. Mott, and J. Sosnin "Sound Mapping, an Assertion of Place", *Proc. Interface '97*.

14. K. Nishimoto, T. Maekawa, Y. Tada, K. Mase, and R. Nakatsu, "Networked Wearable Musical Instruments Will Bring A New Musical Culture", *Proc. ISWC 2001*.

15. *Parkour*, http://www.davidbelle.com/

16. *miniMIXA*, http://www.sseyo.com/minimixa/

17. T. Rueb, *Trace*, http://www.terirueb.net/trace/

18. A. Tanaka, "Mobile Music Making", *Proc. NIME'04*.

19. K. Wodiczko, *Critical Vehicles - Writings, Projects, Interviews* (MIT Press, 1999).

AUTHOR BIOGRAPHIES
LALYA GAYE is a researcher and Ph.D student at the Future Applications Lab, Viktoria Institute (Göteborg, Sweden), working in multidisciplinary projects at the convergence of art, technology, and design. She has a B.Sc in physics from the University of Geneva, Switzerland, an Msc.Eng in electroacoustics from the Royal Institute of Technology KTH in Stockholm, Sweden, and is currently a Ph.D candidate in informatics at the University of Göteborg. Her prototyping-based research explores new territories of personal expression and everyday creativity enabled by ubiquitous and mobile computing, focusing in particular on mobile media for urban space and on computational repurposing of everyday objects. She is also a member of the newly started Pervasive and Locative Arts Network (PLAN), and of sound-art and new media collectives in Göteborg, Sweden.

LARS ERIK HOLMQUIST is the leader of the Future Applications Lab, Viktoria Institute, in Göteborg, Sweden. Before this, he founded and led the PLAY research group from 1997-2001. He received his master’s degree in computer science in 1996, and his Ph.D in informatics in 2000, both at the Göteborg University. His research interests include human-computer interaction, information visualization and ubiquitous computing. He has been member of many international conference committees and published extensively in these research fields. He chaired the international conference on ubiquitous computing UbiComp 2002, and is an associate editor of the Springer journal Personal and Ubiquitous Computing.