Jessica Scarpati
User Experience Researcher, the United States of America
Email: jess.scarpati@gmail.com
Reference this essay: Scarpati, Jessica. “Walking on Code: Of Mobile Locative Storytelling & Augmented Experience.” In Urban Interfaces: Media, Art and Performance in Public Spaces, edited by Verhoeff, Nanna, Sigrid Merx, and Michiel de Lange. Leonardo Electronic Almanac 22, no. 4 (March 15, 2019).
Published Online: March 15, 2019
Published in Print: To Be Announced
ISBN: To Be Announced
ISSN: 1071-4391
Repository: To Be Announced
Abstract
The soundwalk raises questions concerning the sensory and temporal dimensions of space as well as the social conditions of physical and virtual presence. Moreover, it prompts critical thought around relationality, especially insofar as it is inherent to both interfacing and also the publicness of urban space. This essay looks at the soundwalk as a spatially-designed experience and cultural phenomenon, one which specifically draws attention to the nomadic performance of the public and mobile technologies. In short, the soundwalk highlights the theoretical richness of conceptualizing staged events with and through mobility. This essay takes as its central case a soundwalk developed for the Wilhelminapier in Rotterdam. The Points of Departure (2015) soundwalk enabled participants to explore the history of the pier by listening to present-day stories of travel and migration through an app on an Android device. Unique to this soundwalk was the incorporation of the wind as an unpredictable yet determinate factor; by detecting the speed and direction of the wind, a sensor and an algorithm gave rise to permutations of experience.
Keywords: Soundwalk, algorithm, immersion, mobile locative narrative, interface, interfacing
Introduction
The Points of Departure soundwalk is part of a series of works developed for the WilhelminapierExperience Tour, a public-space and online pilot project in Rotterdam, the Netherlands. The Points of Departure event, which staged the prototypes for this project, took place at the Wilhelminapier during March 2015. The soundwalk enabled participants to explore the history of the pier by listening to present-day stories of travel and migration through an app on an Android device. Unique to the Points of Departure soundwalk was the incorporation of the wind as an unpredictable yet determinate factor. The geo-tagged collection of audio recordings that played through the mobile app continually responded to the real-time direction and speed of the wind on the pier by way of an algorithm; the wind direction determined the geographic angle of the location described by the narrator and the wind speed affected the distance. Using the Wilhelminapier as the center, a compass displayed on the mobile app indicated the direction of the wind and the geographical location of the story being relayed. [1]
Described as an installation or performance, the soundwalk developed for the Wilhelminapier can be positioned as advancing the heritage of Nicolas Bourriaud’s relational aesthetics, because of the extent to which it is “beholden to the contingencies of its environment.” [2] Bourriaud’s notion is productive insofar as it allows us to assume a relational vantage point from which we can critically engage with the two notions that underlie this essay, namely: a) the urban interface and b) publicness. Regarding the former, Brandon Hookway’s definition of the interface as a form of relating to technology such that experience comes into being as transformative or transitional is key. [3] For the latter, Chiara Tornaghi’s working conceptualization of publicness as the intersubjective quality of public space which exists not as some a priori thing but as the indistinguishable mix of spatialities—interpreted and used, designed and transgressed—is, likewise, paramount. [4] Following Hookway and Tornaghi, I contend that soundwalks draw attention to the urban interface and to publicness as notions that evoke perceptions of relationality and intersubjectivity that are at once in flux yet able to be traversed.
By regarding the soundwalk as an “active participant in the performance of analysis,” in the way cultural theorist Mieke Bal suggests in various texts, I consider my approach, or method if you like—theoretical object analysis—as one that will be developed through reflection and speculation and with sustained attention to the object. [5] Methodologically, then, the soundwalk is given the space to activate a movement-space of sorts for theoretical thought. It is in this way that I hope to better understand the soundwalk on its own terms.
What, then, are the terms or conditions that make the soundwalk suitable as a theoretical object? First, the Points of Departure soundwalk is an interactive experience that is situated at the intersection of sound, mobile technologies, and the nomadic performance of various agents, which include but are not limited to the participant and the technology at and in hand. Furthermore, as an instance of mobile media art, the soundwalk is an object that invites theoretical investigation of the urban setting, sensory (including the temporal) dimensions of space, and the social conditions of physical and virtual presence—not to mention the emergent experiences that may result from interactions among these factors. [6]
The Points of Departure soundwalk’s relevance to contemporary research on spatiality and temporality in new media is exemplified by the questions it raises. How can one’s experience of space be augmented by code and how may such algorithmic intervention affect one’s sense of spatio-temporal immersion in narrative art, insofar as the stories told relate to one’s immediate surroundings as well as distant others? [7] These questions may otherwise be framed as concerning how algorithms work as interventions in the phenomenology of augmented experience and why augmented experience should necessarily be approached as an experience of immersion. [8] Seeking to answer these questions in the context of the soundwalk represents an attempt, however tenuous, to support Nigel Thrift’s argument that we are increasingly a part of a relative ‘movement-space’ which is open-ended rather than enclosing. [9]
It is through investigating the questions posed above, that we might consider the soundwalk as not only a cultural phenomenon and spatially-designed experience, but also as a ‘theoretical object’—one that presents itself in terms of, produces, and necessitates reflection on theory. [10] As already suggested, the Points of Departure soundwalk may be used to activate a theoretical discussion around a number of concepts. This essay will critically engage with algorithmic intervention and spatio-temporal immersion for the purpose of constructing a theoretical view of the soundwalk’s phenomenological effects on its participants’ experience of both the urban fabric and also the chronotopes of narrative. [11]
By looking at the Points of Departure soundwalk through multiple conceptual lenses, I primarily aim to offer an interpretation of how location-based media involving narrative may reconfigure and even produce both lived and conceived (virtual) space. My analysis articulates some of the ways the combination of mobile technologies and locative storytelling techniques, which combine to form an urban interface, may complicate the ‘soundwalker’s’ perception of the here and now, potentially causing them to simultaneously feel firmly located and uprooted. Moreover, I illuminate how the notion of the interface as a way of relating to technology may be considered as integral to the phenomenology of augmented experience as well as that of publicness. [12] To emphasize the qualities of relationality and intersubjectivity described above, I systematically present theories in fragments and as open-ended rather than enclosing. This is what it might mean to conceptualize the soundwalk with and through mobility.
Thus, I present a reading of the soundwalk as a cultural phenomenon. This reading is fundamentally theoretical; I did not have the opportunity to experience the soundwalk in situ apart from its documentation, which includes the archive of audio recordings that one participating artist kept as an artifact and graciously shared with me one afternoon in Rotterdam nearly a year after the pilot event was staged. Nevertheless, this theoretical reading can help to expand and support the idea that soundwalks can be an avenue toward understanding the lives, events, and alternative histories of people, neighborhoods, and places that are neither included in the official histories nor identified by the historical markers of particular locations. [13] I provide those wanting to theorize or design projects aimed at similar ends with a better understanding of the potential that the interplay between mobile technologies and narrative holds for urban environments. My particular use of the word space is essential to these ends.
Setting the Stage for Space
Throughout this investigation, I reserve space for phenomenological instances of embodied interaction, and I borrow the term embodied interaction from the work of computer scientist Paul Dourish. According to Dourish, understanding that “embodied phenomena are those which by their very nature occur in real time and real space,” makes it possible to posit that “embodiment is the property of our engagement with the world that allows us to make it meaningful,” and, similarly, that “embodied interaction is the creation, manipulation, and sharing of meaning through engaged interaction with artifacts.” [14] For reasons that will become clear throughout this essay, I believe embodied interaction should be extended to the apperception of meaning as well, to account for the ways in which we think about and conceptualize the imagined spaces that may be brought to mind during our interaction with narrative artifacts in lived space.
Embodied interaction, as I have explained it here, coincides with Brandon Hookway’s notion of the interface. According to Hookway, in its operation the interface both “delimits the space that must be traversed for augmentation to occur” and also “opens up onto an exterior spatiotemporality that is only accessible to the augmented subject.” [15] Understanding the interface as such, I propose that interface, as a verb, is to enter into a relation that is augmented (or technologically mediated by engaged interaction with artifacts) and in doing so to conceive of the setting or positioning of elements or properties into relation. Thus, to interface is also to perform the relation as a corporeal event “built on the synchronization of the senses,” including but certainly not limited to hearing. [16] Interpreting the soundwalk as an event of embodied interaction stresses the spatio-temporal nature of interfacing and of navigation. The emphasis this interpretation places on performance is essential, as “the entire body of the user is incorporated in mobility and making space.” [17]
The Production and Augmentation of Space
In the first section below, I explain the problem that the Points of Departure soundwalk sought to solve, and situate the sound walk against a background of critical discourse. I also investigate how an algorithm may act as a means of augmentation. In the second section I discuss how algorithmic intervention affects one’s sense of spatio-temporal immersion.
The Urban Interface: Solving for Space
The Patching Zone is the media and social innovation laboratory in Rotterdam that worked to design the Wilhelminapier Experience Tour pilot project. It set out to find ways to use the surroundings of the urban architecture to define the character of the pier and give it a human dimension. [18] The Wilhelminapier as a site comes not only with a rich history, but also a challenge.
The Wilhelminapier may be considered Rotterdam’s most iconic wharf. From the end of the nineteenth through the early twentieth century, its port served as the departure point for the Holland-America Line and for thousands of migrants who left Europe to start anew in America. Today, the pier is the site of progressive urban redevelopment, and it hosts a cluster of historic warehouses alongside new, state-of-the-art buildings. It has the potential to be a cosmopolitan hub. [19] One perspective from which to look at the redevelopment challenge of the Wilhelminapier concerns the production of space. Despite being privileged as a proper place with a distinctive identity and history, continuous construction and a lack of activities at street level prevent the pier from being fully utilized as a space—the operations and movements that occur between the buildings are disconnected from one another and from the physical location. In other words, passers-by do only that; they pass through without establishing meaning from their route in combination with the history of the pier and the architectural shape-shifting that characterizes it today. In short, it cannot be said that they truly interface with the pier. The soundwalk seeks to remedy this by creating an urban interface—a space defined by publicness that can be traversed and opened up onto exterior spatio-temporalities that are accessible only to the augmented subject. In doing so, it attempts to solve for space by making the public more aware of the relation between the present-day architectural circumstances enabling their movement and the historical impression left by the movement of those who traversed the pier years ago.
Michel de Certeau, among others, has articulated the importance of the connection (and also the distinction) between space and place. For De Certeau, “Space is practiced place. Thus, the street geometrically defined by urban planning is transformed into a space by walkers.” [20] In relation to place, which is constituted by a system of signs, space—the result of what Dourish calls embodied interaction—is dynamic and avoids being defined or fixed. [21] The performative practice of the soundwalk can thus be seen as a practice oriented toward the production of space. Furthermore, in the practice of soundwalking, it is possible to experience “the exchange between the perception of space and place.” [22] This exchange—which, for our purposes, is a product of interfacing—forms the essence of the soundwalk experience. Sound scholar Andra McCartney defines this experience as “an exploration of, and an attempt to understand, the sociopolitical and sonic resonances of a particular location via the act of listening.” [23]
Soundwalks can be used as a research method for exploring the auditory experiences of the location in which the participant is situated. They can even be conducted by ear, perhaps with only a notepad in hand, in a less mediated manner. [24] However, as I have already suggested through my adoption of Dourish’s notion of embodied interaction, they can also incorporate the use of audio devices and mobile technologies to transmit the sounds, or, perhaps, the sonic memories, of disparate locations. It is in such instances, when soundwalks provide an impression of an elsewhere in time or in space, that they more explicitly become storytelling practices. In addition, it is in their performativity that soundwalks stand as evidence for an argument many urban theorists make today, one that I uphold throughout this essay: it is not only bodies but also technologies that continuously shape architectural forms into urban spaces through mediating the rhythms of movement and mobility. [25]
Our interactions with urban environments are increasingly mediated through an array of interfaces, and the Patching Zone actively interrogate this in their soundwalk, which is particularly aimed at orienting, situating, and temporalizing the experiences and histories, even the languages and thoughts, that the Wilhelminapier as a place has gathered over time. [26] With the pervasiveness of mediated navigation in mind, I will now analyze how the algorithmic system designed by the Patching Zone afforded a particular phenomenological experience of space.
Augmenting Space with Algorithms
As previously noted, the Points of Departure soundwalk used the wind as a variable input and digitized it to trigger the pre-recorded audio tales about travel and migration as an output. As a result, the public domain of the pier became a stage for engaged interaction among (at a minimum) four agents or actors: the human agent, the natural phenomenon of the wind, the architectural boundaries, and the mobile-technology system which included the wind sensor, mobile device, algorithmic instruction, and narrative data. [27] To investigate how the algorithm that processed the environmental data fed into the wind sensors acted as a means of augmentation, I begin this section with a discussion of media scholar William Uricchio’s notion of the algorithmic turn and draw comparisons between the effects of augmented reality (AR) and those of the Points of Departure soundwalk. Then I offer a framework through which we can understand the phenomenality generated by algorithmic intervention by discussing the five socio-technical characteristics that sociologist Nigel Thrift identifies as arising out of what he regards as the birth of a new information age: “A transformation, most particularly in the production of space brought about through new practices of organizing, displaying, storing, and communicating information.” [28] These characteristics relate not only to mobile cartographic practices of which interfacing is a part but also to immersive practices in the arts, an umbrella under which the soundwalk will be positioned in the section that follows. [29]
My point of entry for thinking through how algorithms may augment experience is the Points of Departure soundwalk itself. Consider how the variations in the wind and the changes in the code that ran through the listening devices may have altered each participant’s experience. The real-time augmentation of the audio tales was instrumental in creating a unique listening experience in every instance, ensuring not only unpredictability and non-linearity but also a multiplicity of perceptions. In “The Algorithmic Turn,” Uricchio argues that over the past decade, the increased use of location-aware technologies has yielded “increased access to new ways of representing and seeing the world, ways dependent on algorithmic interventions between the viewing subject and the object viewed.” [30] Uricchio approaches the idea of the visual image by adopting a chapter of Jonathan Culler’s Framing the Sign, titled “Semiotics of Tourism” as his theoretical framework and by considering the reworking of subject-object relationships (i.e., the reworking of the phenomenal and the representational). [31] I believe the analytical observations he draws from his case study of location-based augmented reality and to a lesser degree that of the software application Photosynth can be translated to the cultural phenomenon of the soundwalk. Looking at the soundwalk through Uricchio’s observations serves to demonstrate why the Points of Departure soundwalk may be described as an algorithmic intervention.
Keeping in mind the relationality of the wind and algorithm in the Points of Departure soundwalk, I introduce one of Uricchio’s central observations:
The act of mediated looking in AR systems—that is, looking through the camera of the handheld computing device—is also always an algorithmically enabled navigational act. Through the lens of the AR-activated device, there is no such thing as an innocent gaze: the act of gazing and the views consequently seen are transformed into a process of signification as images are laden with particular meanings. [32]
The soundwalk, like an AR system, was an algorithmically-enabled navigational act. The screen of the mobile device used in the soundwalk, contrary to AR systems, did not act as an informational overlay or interface to the world, but the sound did. Uricchio notes that “movement in the AR domain is sited within the physical world, embedded in a multisensory environmental mix of sounds, smells, and presences, but point of view is ultimately limited to that of the viewing subject.” [33] A visual point of view composes the embodied experience of AR; however, an auditory point of view, if we can call it that, takes the forefront against the backdrop of sights, smells, and presences in the embodied experience of soundwalks. In other words, in some soundwalks with listening devices, we are presented with an opportunity to listen to what we don’t see (a real or imagined elsewhere) and see what we don’t listen to (our present environment). [34] It is important to note that although this mutual exclusion of hearing what is not seen and seeing what is not heard applies to the Points of Departure soundwalk, it does not apply to all soundwalks in general. Hypothetically, some soundwalks may use technology to enhance the sounds of the surrounding environment, allowing for what is seen to be better heard. Additionally, in the AR systems cited in “The Algorithmic Turn,” the user’s physical movements and positioning are tracked and algorithmically processed. In the Points of Departure soundwalk, the direction and speed of the wind was tracked instead, but the handheld device still acted as an agent of that tracking and the means by which the participant interfaced with the world in such a way as to enter into an augmented relation to it. [35]
Where the soundwalk diverges from AR, it parallels with Photosynth. Similar to the Photosynth application, which stitches together a three-dimensional image-space from two-dimensional images, the soundwalk—through algorithmic intervention within the database of narratives—becomes an assemblage of many points of view, many authors, and even many instances in time. Uricchio writes, “In Photosynth’s case, while we are aware of seeing through many complementary sets of eyes, the enabler of those viewpoints and of the larger composite remains unseen and out of reach.” [36] The soundwalk, by the same token, uses the wind and a code (both unseen and out-of-reach) to assemble the combination of narrative fragments played for the participant against a constant and ambient background sound. According to narratologist Janet Murray, “[b]y experiencing such interwoven stories as one unit, we can enhance the kaleidoscopic capacity of our minds to imagine life from multiple points of view.” [37] Put another way, it is the process of algorithmic intervention—which is neither wholly the subject’s nor the object’s making—that determines points of access, connections made, and experience gained. [38]
The sound recordings, which became narrative fragments when subjected to the changing direction and speed of the wind, were collected by the Patching Zone during a year of experimenting. During this time, the recordings grew into an archive filled with memories and reflections of travel and migration from places all around the world. The stories chosen for the pilot event consisted of a mix of long and short stories about life-changing experiences people had while traveling abroad. Some recordings describe the first time the narrator left home or felt homesick; all are quiet and intimate, qualities which match the character they take on as transient interventions that follow the direction and speed of the wind. In its fluidity, then, the historically contextualized narrative data serves as a virtual information layer of sorts; it triggers the aforementioned exchange between the perception of space and place by overlaying the Wilhelminapier with new networks of meaning. [39] When the participant, or in our case, the soundwalker, adopts the point(s) of view presented to them, and does so without giving it a second thought, a sense of directness, closeness, and immediacy results in what has been termed absorption. [40] To be absorbed in the narrative sounds of an elsewhere contributes to the apperception of conceived space.
The augmented experience that the soundwalk creates can be likened to an intervention (an augmentation of sorts) for two reasons. First, like the act of looking through the lens of an AR application on a handheld device, the soundwalk is a temporally and spatially bounded experience, one that occurs between other events and areas of urban structure. Second, it acts as an interface, decisively transforming how the participant is able to “construct and/or deconstruct the network of urban meaning” by rendering the urban environment and mobile technologies into components with semantic and theatrical dimensions. [41]
In “The Algorithmic Turn,” Uricchio calls attention to a fundamental transformation of space and time. This transformation makes a phenomenological impression, one which is described in great detail in Thrift’s article “Lifeworld Inc.—And What to Do About It.” [42] What is held in common by these two authors’ mutual confirmation of such a transformation to one’s perception and apperception of space and time—and, arguably, in Hookway’s definition of interface as well—is a view of space (and often time) as being reconfigured by the embodied interaction of human and non-human agents.
For Thrift, in particular, the defining feature of this ‘new regime’ of information and continuous data flows—which demands that we reconstruct our inferences about how the world is connected—is, in Thrift’s words, a ‘generative phenomenality’ that depends on a world of infinite mobilization. “Such an auto-activated world,” Thrift claims, “arises out of five main socio-cultural characteristics which have come into existence over the last thirty years or so, and which, when taken together, point to a reconstruction of the technological unconscious.” [43] These socio-cultural characteristics have much to do with the notion of the interface and with interfacing.
Presently, a summary of the five socio-technical characteristics outlined by Thrift will give way to a discussion of how they may be used to frame the way we understand how one’s experience of space may be algorithmically altered. Notably, all five characteristics are evident in and critical for an analysis of the Points of Departure soundwalk. The five characteristics allude to: (1) a structured continuity, (2) the tactility of the interface or lack thereof, (3) informational overlays, (4) feedback, and (5) non-human agency. In total, these characteristics result in a new phenomenology of algorithmic augmentation, which embraces the performativity of navigation and of interfacing.
The first characteristic is “a structured continuity which always privileges the appearance of movement” and which “gains its phenomenal grip” from ensuring that interfaces (and any differences between interfaces) sink into the background. [44] In the soundwalk, the wind-sensitive algorithm (more so than the physical action of walking) privileges the appearance of movement, and though the stories played are fragmented by the variability of the wind, they pick up where they left off rather than starting anew when the wind returns to an earlier condition, pointing to a structured continuity even amidst fragmentation.
Following from the first characteristic, the second has to do with the nature of the interface used. [45] Thrift refers to not merely touch screens but to interfaces that can continuously augment the physical world with digital information and make any of its surfaces interactive. The nature of the screen-based device used in the soundwalk is one dimension of Thrift’s second characteristic, but as far as interfaces go, it is not so special. Rather, it is the use of sound as a vector for embodiment that stands out when it comes to the soundwalk. Sound, as an invisible interface, mediates the semantic and theatrical dimensions of the urban fabric. This appropriately fits the augmentation highlighted by Thrift.
In a nod to the first and second characteristics, Thrift terms the third awhereness: “the continuity of motion becomes locative as the world is tagged with an informational overlay.” [46] Thrift’s immediate definition of awhereness as quoted here does not sufficiently convey what he means, though. In his illustration of the term and its connection to maps, however, he explains that the ambition is to be able to tag locational identifiers to any unit of content thereby producing what may be called inhabitable maps—those in which location is engineered to produce defined experiences. As far as ‘awhereness’ is concerned, then, we may go so far as to say that through location-tagged narratives, the soundwalk extends spatial awareness to geographically remote places and links previously isolated moments and spaces, fostering an experience of spatial and temporal coincidence.
As already explained, the soundwalk extends spatial awareness by way of constant feedback, the fourth characteristic. Constant feedback has double significance to Thrift. On the one hand, it is corporeal, encompassing the extension of “physical geography of bodily interaction” to technologies and the production of new forms of mindfulness coming from those same technologies in turn. [47] On the other, constant feedback makes possible “interactive compositions” in media, in which improvisatory technologies allow users active co-construction through live algorithms. [48] In the case of the Points of Departure soundwalk, it is not the participant interacting with the algorithm to create the composition. Instead, the environment—yet another non-human agent—takes responsibility for co-creation.
The fifth and last characteristic that Thrift identifies brings forward part of the fourth, that is, “that cognition becomes even more of a joint experience between persons and things.” [49] This feature, calling upon arguments reminiscent of actor-network theory, emphasizes that agency increasingly presides in non-human entities. [50] As has become evident in each of the previous characteristics, the soundwalk prompts re-cognition of the environment through embodied interaction between participants and non-human artifacts or agents, providing the opportunity for all those agents involved to interface against the backdrop this new information age. As Thrift explains, drawing from the philosopher Alfred North Whitehead, it is as if more things are able to become able. [51]
The attention I have paid to the soundwalk thus far has been limited to the impact of the algorithm on one’s experience of space. Describing the affinities among the observations made by Uricchio and Thrift has demonstrated that engaged interaction with mobile-technologies, their interfaces, and their algorithms “produce [a sense] of immersion which both limits (through the selective engagement of attention) and expands what can be sensed and how it can be sensed through careful attention to movement through space.” [52] Until now I have conceptualized this experience of space as one produced by embodied interaction. However, I have not yet addressed the sense of immersion produced by the soundwalk, which has a phenomenality of its own as an instance of algorithmic augmentation. The definition of immersion that I put forward here and expand on in the following section also comes from Thrift. For the purpose of this essay, I define immersion as a “compelling sense of always being already there” and as an “awareness of something that feels like it is already known.” [53] In the context of this essay, that something has to do with the intrinsic connectedness of our temporal and spatial relationships with technics, with space, and in a sense, with the chronotopes of narrative.
Immersion in Mobile Locative Narratives
At the start of this essay, I categorized the Points of Departure soundwalk as an instance of mobile media storytelling, “a mode of storytelling that blends digital media on mobile devices with physical environments.” [54] From a slightly different point of view, we may also classify the collection of audio narratives as mobile locative narratives. Mobile locative narratives tie stories to a location in the physical environment with the help of mobile devices, and by extension, they also tend to reflect on that specific location. [55] In the Points of Departure project, the narratives are not so much binding as they are emergent vis-à-vis the algorithmic logic already described. Nevertheless, they are contextual, and they create an experience of (narrative) immersion.
In this section, I will analyze how algorithmic intervention affects one’s sense of spatio-temporal immersion. Accordingly, I begin by focusing on how the nature of the stories told as well as the intervention of algorithms in soundwalks can affect, or otherwise play with, the subject’s sense of spatio-temporal immersion in the narratives, “by collapsing the boundary between the digitally mediated storyworld[s] and the physical world.” [56] Then, by reflecting on how the use of the mobile technology in the Points of Departure soundwalk was used to envelope geographically distant places within a specific spatial locus, it will become possible to theorize how the soundwalk toyed with ‘awhereness’ as well as with the boundaries between place and (lived and conceived) space. This theorization will, ultimately, serve the primary goal of this essay as articulated in the introduction: to offer an interpretation of how location-based media involving narrative may reconfigure and even produce both lived and conceived (virtual) space.
Creating a Sense of Immersion
For articulating the soundwalk’s ability to bring about a sense of immersion on the level of the storyworlds, I refer to literary scholar Marie-Laure Ryan, who—when compared to Thrift— speaks more definitively of narrative:
One of the most variable parameters of narrative art is the imaginative distance between the position of the narrator and the addressee and the time and place of the narrated events. Spatio-temporal immersion takes place when this distance is reduced to near zero. [57]
This explanation comes close to the earlier conception of immersion put forward, to the extent that reducing imaginative distance to ‘near zero’ implies a feeling of ‘always being already there.’ There is a difference between these phrases worth pointing out. One is purely spatial (near zero) while the other (already) is also temporal. According to Ryan, there are a number of functions which stories use to pull temporally or spatially distant objects “into the theater of the mind” to “coax the imagination into simulating sensory perception.” [58] The fabrication of floating consciousness and the use of proper names are the two functions developed by Ryan that are most relevant to the stories relayed in Points of Departure soundwalk.
In the narratives recorded for the soundwalk, the narrators are the characters in their own stories. They do not introduce themselves, instead, they give a first-person perspective of their experiences traveling or migrating. French literary theorist Gérard Genette calls this point-of-view paradigm in narrative internal focalization.[59] Internal focalization corresponds functionally to the fabrication of floating consciousness. [60] Ryan argues that when internal focalization takes the form of direct personal memories, it “[enables] readers to construct a precise map of the textual world and to visualize the changing environments as the characters move from location to location.” [61] The fabrication of floating consciousness produces a minimal difference between staged and daily reality, as the mobile audio device produces a secret theater:
The isolated listener is corporeally affected by an acoustic, invisible source, which causes (slight) changes in pace and behaviour. The soundscape makes the walk ‘more poetic and more dramatic’ in Shuhei Hosakawa’s words, and turns all passers-by into figures on a theatre stage. [62]
This minimal difference that comes from the illusion of a secret theater is a facet of displacement and enables the soundwalk to bring geographically distant locations into the proximity of a newly conceived space.
Publicness: Traveling Between Lived and Conceived Space
Any given participant of the Points of Departure soundwalk may have experienced immersion in the storyworlds conveyed or immersion in the physical space. This does not withstand the possibility that they might have experienced both levels of immersion at once or experienced a continuous shift from one to the other. Additionally, it is not unreasonable to speculate that some of the soundwalk participants could have imagined walking with or as the narrators as they told their stories from their respective locations around the globe. As sound and digital media researcher John Barber suggests, “[a]ttributes of the narrative soundscape, particularly the human voice (as in oral tradition), may promote a unique sense of place and connection.” [63] This sense of absorption, which counterbalances that of displacement, may be attributed to the function of proper names.
It is important to recall that the soundwalk in question did not afford a linear listening experience; the intervention of the algorithm made the development of textual geography—that is, the landscapes linguistically illustrated by the story—nearly impossible to achieve with lengthy descriptions. However, Ryan finds the use of proper names to be an efficient way to create a sense of place in lieu of such descriptions. She argues that the most complete forms of spatial immersion happen when the reader or listener develops a sense of being present on the scene of conveyed events, and names conjure up such scenes. [64] Names of countries are used to varying degrees in the stories of the soundwalk, but the compass on the mobile app used during the soundwalk always displayed the name of the place being recalled along with the direction from where the wind and story were coming.
While the production of space is of central concern, there is value in attending to the role the compass had in relaying the proper names of places. The compass was not an insignificant feature, in fact, it is possible to argue that, with the help of the algorithm and its status as a material metaphor, it allowed for interaction with both the physical world and also the digitally mediated storyworld. [65] Calling the existence of distant locations to the attention of the participant, the combination of the visual representation of the compass as mediating interface and the aural stories as augmented soundscape reinforced the power that places have to gather not only things but also experiences, histories, languages, and thoughts in their midst. [66] At the same time, the compass made possible a conscious experience of the sensation of wind by highlighting its direction, adding to the sensory inputs of the soundwalk and making possible a feeling of ‘always being already there.’ Through an overlay of different sensory inputs, the Points of Departure soundwalk enabled an experience of embodiment and a sense of immersion in a perhaps indistinguishable mix of spatialities, both lived and conceived.
According to Barber, “[w]hen combined with the information-carrying capacity of other senses, the aural provides the ability to attach multiple narratives, in multiple forms, to a specific location.” [67] This process of signification is not a simple one, especially in the case of the Points of Departure soundwalk where the stories emphasized distance, while the intimacy of the narrator’s voice and kinesthetic engagement of the participant with the landscape produced an effect of presence. Presence, as it is used here, implies a tension between the soundwalker’s private, imagined landscape and his embodiment in the here and now. In calling attention to this tension, the soundwalk afforded a sense of publicness as well. That is to say, it emphasized that the intersubjective quality of public space does not exist as some a priori thing but as the indistinguishable mix of spatialities interpreted and used, designed and transgressed.
Concluding Remarks
In the Points of Departure soundwalk, mobile technology has the effect of producing a secret theatre, as it is primarily used as a means of borrowing the conceived space of geographically distant others. The soundwalk represents, in Thrift’s terms, “an explicit return to a kind of nomadism which no longer privileges fixed territory as necessary to produce effects but which does not therefore think that the attachments of territory are somehow unimportant.” [68] Mobile locative narratives afford navigation from one consciousness to another. In the Points of Departure project, the distributed events that take the form of the audio stories come together into the singular located occurrence of the soundwalk, and, in doing so, can temporarily displace the participant’s sense of presence and embodiment in lived space in exchange for a heightened awareness of conceived space. [69] Nevertheless, as Thrift implies, this does not mean that a sense of presence, of territorial attachment, or of place is unimportant. Instead, the assemblage of voices—an archive of identities coinciding with space—colors local experience turning the Wilhelminapier into an embodied place, a public space, by Tonaghi’s terms. As for the ‘soundwalker,’ they take on an active, connected role in this process via the act of interfacing, that is, of relating to the history of the pier and its current architectural form through which they traverse.
By using mobile technologies to alter our experience of physical space through the “creation of hybrid realms that incorporate physical, digital, and represented spaces,” [70] soundwalks enable locative engagement that is at once sonic, tactile, and kinesthetic. If we go so far as to call the soundwalk a medium, then it is first and foremost a spatial medium. Like the mobile media that support them, they are also spatially flexible and “thus are uniquely equipped to engage with narratives about spaces and places.” [71] Through soundwalks, locations can begin to present themselves in new ways, as an indistinguishable mix of spatialities—interpreted and used, designed and transgressed. In this way, soundwalks enable places and spaces to take on air of publicness. They are one medium through which an urban interface may be formed, and therefore, they are indicative of transformation or augmentation. When they incorporate factors that are to be processed by code in real time, soundwalks become algorithmic objects—sequences of iterations that come into being as they move through time. [72]
As a final thought, to accept spatio-temporal immersion as an effect of algorithmic intervention and the interaction between the various agents involved in the performance of the Points of Departure soundwalk, it is important to acknowledge that the soundwalk was produced at a particular time and place for a historically and culturally specific participant—one who is accustomed to entering into relation with technology and to walking on and amidst code. My analysis suggests that mobile technologies and their algorithms assume a central role in making the Points of Departure soundwalk part of the new information age of augmentation, of interfacing, and of ubiquitous computing, as described by Uricchio, Thrift, Hookway and numerous others. If it has not already, this age may come to be defined by the ways in which interfacing—even sonically, like the soundwalk—with artifacts capable of algorithmic intervention can reconfigure embodied space. Today, it is not only bodies but also technologies that continuously shape architectural forms into urban spaces through mediating the rhythms of movement and mobility.
Author Biography
Jessica Scarpati earned her RMA in Media and Performance Studies from Utrecht University in 2017. She holds a Bachelor degree from Emerson College. Her research interests lie at the intersection of media archaeology and posthumanism. Her approach most often combines theoretical analysis and ethnography. This is best demonstrated by her master’s thesis, which proposes that Walter Benjamin’s notion of aura, a form of perception that invests a phenomenon with the ability to look back, be reactivated for the purpose of affirming the vitality exhibited by robots staged in performative situations. Jessica currently resides in Boston, where she is a practicing experience design researcher.
Notes and References
[1] Anne Nigten, ed., Real Projects for Real People (Rotterdam: The Patching Zone, 2015).
[2] Nicolas Bourriaud, “Relational Aesthetics,” in Participation: Documents of Contemporary Art, ed. Claire Bishop (Cambridge, MA: MIT Press, 1998), 160, in reference to Mimi Sheller, “Vital Methodologies: Live Methods, Mobile Art, and Research-Creation,” in Non-Representational Methodologies: Re-Envisioning Research, ed. Phillip Vannini (London and New York, NY: Routledge, 2015), 134.
[3] Branden Hookway, Interface (Cambridge, MA and London: MIT Press, 2014).
[4] Hookway, Interface, 1-6; Chiara Tornaghi, “The Relational Ontology of Public Space and Action-oriented Pedagogy in Action,” in Public Space and Relational Perspectives: New Challenges for Architecture and Planning, eds. Chiara Tornaghi and Sabine Knierbein (London and New York, NY: Routledge, 2014), 24.
[5] Mieke Bal, Travelling Concepts in the Humanities (Toronto: University of Toronto Press, 2002); Mieke Bal, “Visual Essentialism and the Object of Visual Culture,” Journal of Visual Culture2, (2003): 24.
[6] Mobile media storytelling may be defined as “a mode of storytelling that blends digital media on mobile devices with physical environments.” See Brett Oppegard and Dene Grigar, “The Interrelationships of Mobile Storytelling,” in The Mobile Story: Narrative Practices with Locative Technologies, ed. Jason Farman (London and New York, NY: Routledge, 2014), 17.
[7] The concepts algorithmic intervention and spatio-temporal immersion are borrowed from William Uricchio and Marie-Laure Ryan, respectively. See William Uricchio, “The Algorithmic Turn,” Visual Studies 26, no. 1 (2011): 25-35; Marie-Laure Ryan, Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media (Baltimore, MD: Johns Hopkins University Press, 2011), 120-139.
[8] My utilization of the term augmented proceeds from the work of Lev Manovich and Nanna Verhoeff, who conceptualize augmented space as physical space that is overlain with layers of data. See Lev Manovich, The Language of New Media (Cambridge, MA: MIT Press, 2001); Nanna Verhoeff, Mobile Screens: The Visual Regime of Navigation (Amsterdam: Amsterdam University Press, 2012), 124.
[9] Nigel Thrift, “Movement-Space: The Changing Domain of Thinking Resulting from the Development of New Kinds of Spatial Awareness,” Economy and Society 33, no. 4 (2004): 592.
[10]Yve-Alain Bois, et al., “A Conversation with Hubert Damisch,” October 85 (Summer, 1998): 8-9.
[11] Chronotope here refers to the concrete whole formed by the “intrinsic connectedness of temporal and spatial relationships that are artistically expressed in literature.” See M. M. Bakhtin, “Forms of Time and of the Chronotope in the Novel,” in Narrative Dynamics: Essays on Time, Plot, Closure, and Frames, ed. Brian Richardson (Columbus, OH: Ohio State University Press, 2002), 15.
[12] Hookway, Interface, 1.
[13] Daniel Makagon and Mark Neumann, Recording Culture: Audio Documentary and the Ethnographic Experience (Thousand Oaks, CA: SAGE Publications, 2008).
[14] Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction (Cambridge, MA: MIT Press, 2001), 126.
[15] Hookway, Interface, 18.
[16] Verhoeff, Mobile Screens, 168.
[17] Ibid., 150.
[18] The Wilhelminapier Experience Tour is part of the international Live Transmission project; the pilot project was the first concrete result of the collaboration among the five cultural partners of the Wilhelminapier: the SKVR | Beeldfabriek, LantarenVenster, LP2, Nederlands Fotomuseum, and the Luxor Theatre. See Nigten, Real Projects, 16.
[19] Ibid., 15.
[20] Michel De Certeau, The Practice of Everyday Life, trans. Steven Rendall (Berkeley, CA: University of California Press, 1984), 117.
[21] Ibid.
[22] Alessandro Altavilla and Atau Tanaka, “The Quiet Walk: Sonic Memories and Mobile Cartography,” in Proceedings of the 9th sound and music computing conference (2012): 157-162.
[23] Ibid.; Andra McCartney, “Soundscape Works, Listening, and the Touch of Sound,” in Aural Cultures, ed. Jim Drobnick (Toronto: YYZ Books, 2004).
[24] McCartney, “Soundscape Works,” 180; For more on soundwalking as method, see Andra McCartney, “Soundwalking: Creating Moving Environmental Sound Narratives,” in The Oxford Handbook of Mobile Music Studies, eds. Sumanth Gopinath and Jason Stanyek, vol. 2 (Oxford: Oxford University Press, 2014).
[25] Adriana de Souza e Silva, “From Cyber to Hybrid: Mobile Technologies as Interfaces of Hybrid Spaces,” Space and Culture 9, 3 (2006): 261-278; Gretchen Schiller and Sarah Rubridge, eds., Choreographic Dwellings: Practising Place (London and New York, NY: Palgrave MacMillan, 2014).
[26] De Certeau, The Practice of Everyday Life, 117; McCartney, “Soundscape works.”
[27] I use agent and actor as interchangeable concepts. See Bruno Latour, Reassembling the Social: An Introduction to Actor-Network Theory (Oxford: Oxford University Press, 2005).
[28] Nigel Thrift, “Lifeworld Inc.—and what to do about it,” Environment and Planning D: Society and Space 29, no. 1 (2011): 2.
[29] Verhoeff, Mobile Screens, 133.
[30] Uricchio, “The Algorithmic Turn,” 25; It is not my intention to enter into a debate on the extent to which images viewed and sounds heard can be equated, so for the purpose of this essay, I will operate under the premise that they are both sensory inputs and just as visual representations have become subject to algorithmic intervention, so too have auditory ones.
[31] See Jonathan Culler, “The Semiotics of Tourism,” in Framing the Sign: Criticism and Its Institutions (Norman, OK: University of Oklahoma Press, 1990).
[32] Uricchio, “The Algorithmic Turn,” 32.
[33] Ibid., 33, emphasis mine.
[34] Shuhei Hosokawa, “The Walkman Effect,” Popular Music 4 (1984): 167.
[35] Cf. Uricchio, “The Algorithmic Turn,” 32.
[36] Ibid., 31.
[37] Janet H. Murray, Hamlet on the Holodeck: The Future of Narrative in Cyberspace (Cambridge, MA: MIT Press, 1998), 160-161.
[38] Uricchio, “The Algorithmic Turn,” 33.
[39] Ed Finn, What Algorithms Want: Imagination in the Age of Computing (Cambridge, MA, and London: MIT Press, 2017), 52.
[40] Maaike Bleeker, “Absorption and Focalization,” Performance Research 10, no. 1 (2005): 53.
[41] Hosokaway, “The Walkman Effect,” 178; Christopher R. Balme, “Audio Theatre: The Mediatization of Theatrical Space,” in Intermediality in Theatre and Performance, eds. Freda Chapel and Chiel Kattenbelt (Amsterdam: Radopi, 2006), 118.
[42] See Thrift, “Lifeworld Inc.”
[43] Ibid., 5.
[44] Ibid.
[45] Ibid.
[46] Ibid., 6.
[47] Ibid., 8.
[48] Ibid., 8.
[49] Thrift, “Lifeworld Inc,” 8-9.
[50] See Latour, Reassembling the Social.
[51] Thrift, “Lifeworld Inc.,” 9.
[52] Ibid., 14.
[53] Ibid., 15.
[54] Oppegard and Grigar, “The Interrelationships of Mobile Storytelling,” 17.
[55] Jeff Ritchie, “The Affordances and Constraints of Mobile Locative Narratives,” in The Mobile Story: Narrative Practices with Locative Technologies, ed. Jason Farman (London and New York, NY: Routledge, 2014), 61.
[56] Ibid.; For more on the term storyworld, see David Herman, Story Logic: Problems and Possibilities of Narrative (Lincoln, NE: University of Nebraska Press, 2002), 16.
[57] Ryan, Narrative as Virtual Reality, 130.
[58] Ibid., 122.
[59] The Interdisciplinary Center for Narratology, “Focalization,” last modified August 4, 2011, http://wikis.sub.uni-hamburg.de/lhn/index.php/Focalization.
[60] Ryan, Narrative as Virtual Reality, 133.
[61] Ibid., 128.
[62] Liesbeth Groot-Nibbelink, “Nomadic Theatre: Staging Movement and Mobility in Contemporary Performance” (PhD diss., Utrecht University, 2015), 65.
[63] John Barber, “Walking-Talking: Soundscapes, Flaneurs, and the Creation of Mobile Media Narratives,” in The Mobile Story: Narrative Practices with Locative Technologies, ed. Jason Farman (London and New York, NY: Routledge, 2014), 101.
[64] Ryan, Narrative as Virtual Reality, 128.
[65] Katherine Hayles proposes the term material metaphor to foreground the idea that a digital symbol (whose physical properties and historical usages structure our interactions with it) can actually make things happen in the real world; See Katherine Hayles, Writing Machines (Cambridge, MA: MIT Press, 2002).
[66] McCartney, “Soundscape works.”
[67] Barber, “Walking-Talking,” 101.
[68] Thrift, “Lifeworld Inc.,” 19.
[69] Sally Jane Norman, “Anatomies of Live Art,” in Anatomy Live: Performance and the Operating Theatre, ed. Maaike Bleeker (Amsterdam: Amsterdam University Press, 2008), 189.
[70] Oppegard and Grigar, “The Interrelationships of Mobile Storytelling,” 28.
[71] Jason Farman, ed., The Mobile Story: Narrative Practices with Locative Technologies (London and New York, NY: Routledge, 2014), 8.
[72] Finn, What Algorithms Want, 131.
Bibliography
Altavilla, Alessandro, and Atau Tanaka. “The Quiet Walk: Sonic Memories and Mobile Cartography.” In Proceedings of the 9th Sound and Music Computing Conference, 157-162.2012.
Bakhtin, M. M. “Forms of Time and of the Chronotope in the Novel.” In Narrative Dynamics: Essays on Time, Plot, Closure, and Frames, edited by Brian Richardson, 15-24. Columbus, OH: Ohio State University Press, 2002.
Bal, Mieke. Travelling Concepts in the Humanities. Toronto: University of Toronto Press, 2002.
Bal, Mieke. “Visual Essentialism and the Object of Visual Culture.” Journal of Visual Culture2, (2003): 5-32.
Balme, Christopher R. “Audio Theatre: The Mediatization of Theatrical Space.” In Intermediality in Theatre and Performance, edited by Freda Chapel and Chiel Kattenbelt, 117-124. Amsterdam: Radopi, 2006.
Barber, John. “Walking-Talking: Soundscapes, Flaneurs, and the Creation of Mobile Media Narratives.” In The Mobile Story: Narrative Practices with Locative Technologies, edited by Jason Farman, 95-110. London and New York, NY: Routledge, 2014.
Bleeker, Maaike. “Absorption and Focalization.” Performance Research 10, no. 1 (2005): 48-60.
Bois, Yve-Alain, Denis Hollier, Rosalind Krauss, and Hubert Damisch.“A Conversation with Hubert Damisch.” October 85 (Summer, 1998): 3-17.
Bourriaud, Nicolas.“Relational Aesthetics.” In Participation: Documents of Contemporary Art, edited by Claire Bishop, 160-171. Cambridge, MA: MIT Press, 1998.
Culler, Jonathan. “The Semiotics of Tourism.” In Framing the Sign: Criticism and Its Institutions, 1-10. Norman, OK: University of Oklahoma Press, 1990.
De Certeau, Michel. The Practice of Everyday Life. Translated by Steven Rendall.Berkeley, CA: University of California Press, 1984.
De Souza e Silva, Adriana. “From Cyber to Hybrid: Mobile Technologies as Interfaces of Hybrid Spaces.” Space and Culture 9, no. 3 (2006): 261-278
Dourish, Paul. Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press, 2001.
Farman, Jason, ed. The Mobile Story: Narrative Practices with Locative Technologies .London and New York, NY: Routledge, 2014.
Finn, Ed. What Algorithms Want: Imagination in the Age of Computing. Cambridge, MA, and London: MIT Press, 2017.
Groot-Nibbelink, Liesbeth. “Nomadic Theatre: Staging Movement and Mobility in Contemporary Performance.” PhD diss., Utrecht University, 2015.
Hayles, Katherine. Writing Machines. Cambridge, MA: MIT Press, 2002.
Herman, David. Story Logic: Problems and Possibilities of Narrative. Lincoln, NE: University of Nebraska Press, 2002.
Hookway, Branden. Interface. Cambridge, MA and London: MIT Press, 2014.
Hosokawa, Shuhei. “The Walkman Effect.” Popular Music 4 (1984): 165-180.
Latour, Bruno. Reassembling the Social: An Introduction to Actor-Network Theory. Oxford: Oxford University Press, 2005.
Makagon, Daniel, and Mark Neumann. Recording Culture: Audio Documentary and the Ethnographic Experience. Thousand Oaks, CA: SAGE Publications, 2008.
Manovich, Lev. The Language of New Media. Cambridge, MA: MIT Press, 2001.
McCartney, Andra. “Soundscape Works, Listening, and the Touch of Sound.” In Aural Cultures, edited by Jim Drobnick, 179-185. Toronto: YYZ Books, 2004.
McCartney, Andra. “Soundwalking: Creating Moving Environmental Sound Narratives.” In The Oxford Handbook of Mobile Music Studies, edited by Sumanth Gopinath and Jason Stanyek, vol. 2, 212-237. Oxford: Oxford University Press, 2014.
Murray, Janet H. Hamlet on the Holodeck: The Future of Narrative in Cyberspace.Cambridge, MA: MIT Press, 1998.
Nigten, Anne, ed. Real Projects for Real People. Rotterdam: The Patching Zone, 2015.
Norman, Sally Jane. “Anatomies of Live Art.” In Anatomy Live: Performance and the Operating Theatre, edited by Maaike Bleeker, 187-204. Amsterdam: Amsterdam University Press, 2008.
Oppegard, Brett, and Dene Grigar. “The Interrelationships of Mobile Storytelling.” In The Mobile Story: Narrative Practices with Locative Technologies, edited by Jason Farman, 17-33. London and New York, NY: Routledge, 2014.
Ritchie, Jeff. “The Affordances and Constraints of Mobile Locative Narratives.” In The Mobile Story: Narrative Practices with Locative Technologies, edited by Jason Farman, 53-67. London and New York, NY: Routledge, 2014.
Ryan, Marie-Laure. Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Baltimore, MD: Johns Hopkins University Press, 2011.
Schiller, Gretchen, and Sarah Rubridge, eds. Choreographic Dwellings: Practising Place. London and New York, NY: Palgrave MacMillan, 2014.
Sheller, Mimi. “Vital Methodologies: Live Methods, Mobile Art, and Research-Creation.” In Non-Representational Methodologies: Re-Envisioning Research, edited by Phillip Vannini, 130-145. London and New York, NY: Routledge, 2015.
The Interdisciplinary Center for Narratology. “Focalization.” Last modified August 4, 2011. http://wikis.sub.uni-hamburg.de/lhn/index.php/Focalization.
Thrift, Nigel. “Lifeworld Inc.—and what to do about it.” Environment and Planning D: Society and Space29, no. 1 (2011): 5-26.
Thrift, Nigel. “Movement-Space: The Changing Domain of Thinking Resulting from the Development of New Kinds of Spatial Awareness.” Economy and Society 33, no. 4 (2004): 582-604.
Tornaghi, Chiara. “The Relational Ontology of Public Space and Action-oriented Pedagogy in Action.” in Public Space and Relational Perspectives: New Challenges for Architecture and Planning, edited by Chiara Tornaghi and Sabine Knierbein, 17-41. London and New York, NY: Routledge, 2014.
Uricchio, William.“The Algorithmic Turn.” Visual Studies 26, no. 1 (2011): 25-35.
Verhoeff, Nanna. Mobile Screens: The Visual Regime of Navigation. Amsterdam: Amsterdam University Press, 2012.