William Uricchio
Prof. Dr., Comparative Media Studies, MIT, Massachusetts, USA
Email: uricchio@mit.edu
Web: http://williamuricchio.com
Reference this essay: Uricchio, William. “Augmenting Reality: The Markers, Memories, and Meanings Behind Today’s AR.” In Urban Interfaces: Media, Art and Performance in Public Spaces, edited by Verhoeff, Nanna, Sigrid Merx, and Michiel de Lange. Leonardo Electronic Almanac 22, no. 4 (March 15, 2019).
Published Online: March 15, 2019
Published in Print: To Be Announced
ISBN: To Be Announced
ISSN: 1071-4391
Repository: To Be Announced
Abstract
Humans have long ‘marked’ the world, cities included, augmenting their surroundings with traces of their experience. They have learned to read the signs inscribed on the world’s surface, whether in search of prey or precedent. And they have sorted out how to give those marks coherence, whether as narrative or insight. Publically deployed, discovery-based, or even personal, augmentation’s inherently dialogic character requires reality for its work. This essay considers today’s AR technologies in terms of these more deeply embedded practices of augmentation, particularly as they play out as interfaces in urban places. It explores underlying continuities between earlier augmentation practices and our own, suggests strategies that can usefully inform our deployment of AR technologies, and points to semiotic reciprocities that can help AR move from the gadget du jour to an ally in a profoundly human conversation.
Keywords: Augmented Reality, AR, spatial narrative, memory palaces, tourism, cities, places and spaces, public memory, emerging taxonomies
Introduction
Several years ago, when writing an essay about algorithms and the changing status of the image, I found inspiration in an essay by Jonathan Culler on the semiotics of tourism. [1] Culler traced the processes by which cultural attractions are marked as signs, looking beyond them as simply responsive to and inscribed within an economic order, and saw an opportunity to explore the persistent and ubiquitous semiotic mechanisms that give social orders their coherence. ‘Coherence’ in this sense is a slippery concept and enjoys a certain taken-for-grantedness, underpinning as it does the fabric of associations, values, and perceptions that enable our navigation of the social world. We have well-developed traditions for looking at language and ideology as sites of social order and means of navigation; and a generation of now-canonized scholars such as Barthes, Bourdieu, Hoggart, and Williams have helped us to explore their operations in fashion, manners, and everyday culture. But Culler’s added value was to tease out coherence as a temporally and spatially pervasive process, one that facilitates the navigation of the physical world while situating its meanings and historical resonances. I will continue to draw on his insights in this essay on urban interfaces in the form of the annotation and augmentation of everyday interactions with our spatial environment.
One of the semiotic strategies that Culler discusses is the ‘marker.’ Drawing on Dean MacCannell’s The Tourist, Culler defines a marker as any kind of information or representation,
…that constitutes a sight as a sight: by giving information about it, representing it, making it recognizable. Some are ‘on-site’ markers, such as plaques telling that ‘George Washington slept here’ or that this vial of dust comes from the moon. Some are mobile markers, such as pamphlets and brochures designed to draw people to the site, give information at the site, and serve as souvenirs or representations off the site…. The proliferation of markers frames something as a sight for tourists. [2]
Culler’s essay and MacCannell’s insights go a long way towards explaining how value and meaning are created by collapsing temporalities, by overlaying a site or object in the present with a reference to its past, and more generally, by a process of indication. The process of generating value by overlaying, framing, and indicating meaning is a familiar and a complicated one, with its own spatial and temporal logics. Dynamic and contingent rather than intrinsically bound to particular utterances and artefacts, Culler’s ‘markers’ have much in common with interfaces, especially as the term is deployed in this collection of essays, capable of producing what Alexander Galloway has described as “a generative friction between different formats.” [3]
This essay will offer a few different passes at, broadly speaking, ‘markers’ and acts of indication that occur through the conjunction of space and memory and that result in the semiotic augmentation of the world. This augmentation, whether technological (AR) or perceptual (informed ways of seeing), functions as an interface. Narrative is important in this process, both as a pre-existing pathway marked by established signs (a framing strategy that enables Culler’s ‘ubiquitous semiotic mechanisms’), but also as an experiential process that creates both signs and coherence for certain agents. Technique and technology loom large in this story, which will begin with the memory palaces of the ancient Greeks and Romans and end up with today’s ongoing developments in augmented reality, broadly defined. At a moment when terms such as ‘virtual reality,’ ‘mixed reality,’ and ‘augmented reality’ are being tossed around as much as marketing banter as aspiration, it may help to think more broadly and even historically about the endeavours underlying these claims, and in the case of this essay, the augmentation of urban reality – and its resulting interfaces – in particular.
Stepping Back…
We know, of course, that the physical overlays of the world in the form of advertisements, public signage, architecture, statuary, and plaques have a function: to inscribe places with meaning, as MacCannell and Culler suggest. And they have a deep and variegated history. Consider the ancient Via Dolorosa in Jerusalem, where a long – and sometimes awkwardly changing – tradition has physically marked out the stations of Christ’s final days in the form of churches, sacred stones, signage, tourist shops, and maps. These markers augment an otherwise relatively ordinary place, endowing it with historical meaning, steeping it with religious aura, and enabling visitors to wander through (and for extreme devotees, even enact) a spatial narrative. In this case, the markers overlay what believers take to be the literal spots where Christ’s last journey occurred. By contrast, gothic cathedrals across Europe (as well as their global cousins) offer a geographically-agnostic architectural constellation where elements of the Christian narrative undergird the orientation of the altar, the cruciform floor plan, and the statuary and windows (all to the accompaniment, if the visitor’s timing is right, of a multi-media wrap including sacred words and song, scents, and performance). These markers recall key moments from the same basic narrative as inscribed on the Via Dolorosa, referencing its historical details and radiating its religious aura, even if far removed from the geographical location where the original events played out. Although I have here invoked as an example physical spaces and places whose narrative coherence depends upon a pre-existing ur-text (the New Testament), the larger process of reading markers, and through that reading, creating the experience of a participatory and spatially dispersed narrative, is likely as old as our species.
In his “Clues: Roots of an Evidentiary Paradigm,” Carlo Ginzburg makes a bold suggestion: “…Perhaps the actual idea of narration (as distinct from charms, exorcisms, or invocation) may have originated in a hunting society, relating the experience of deciphering tracks.” [4] Although acknowledging this as “an obviously undemonstrable hypothesis,” Ginzburg makes the case for linking “venatic deduction” to narration by examining language, where it is deeply embedded. [5] Ginzburg goes on to say,
…we are struck by the undeniable analogies between the venatic model just discussed and the paradigm implicit in the Mesopotamian divination texts, which began to be composed in the third millennium B.C. Both presuppose the minute investigation of even trifling matters, to discover the traces of events that could not be directly experienced by the observer. Excrement, tracks, hairs, feathers in one case; animals’ innards, drops of oil on the water, heavenly bodies, involuntary movements of the body, in the other. [6]
Despite their divergent epistemological stakes, these two traditions – hunting and divining – share such operations as analysis, comparison, and classification in their attempts to decipher the variously construed ‘markers’ and clues that give the unseen world its meanings.
Ginzburg’s linkage of narration with the age-old practice of hunting, with making visible the invisible and drawing meaning from signs no matter how tenuous, is intriguing. His discussion focuses on what we might term reading practices and the deciphering of signs in a manner that reveals a coherent series of events. In fact, elements of this strategy were formalized by the ancient Greeks and Romans who used similar insights to develop a memory system in the form of the memory palace, literalizing a spatial metaphor as a storage and retrieval system for ideas, and creating a virtual writing and reading system for narratives of discovery. The previously mentioned gothic cathedrals and Via Dolorosa do this in a declamatory, intersubjective way: physical markers punctuate correlations between a location and well-known, pre-existing narrative. Ginzburg’s narrative, by contrast, while also potentially intersubjective, can only be extrapolated by ‘those in the know’ who are capable of deciphering signs that are significant to them – but meaningless to the uninformed. The domain of the memory palace, however, is purely subjective, a pre-formatted and well-rehearsed mental-spatial configuration populated with (changeable) visual memory markers – the more striking, the better: Ginzburg’s “excrement, … animals’ innards, … heavenly bodies…” [7] The performance of space that defines interactions with the palace draws on an ambulatory metaphor for its enactments of malleable association, pluriform meaning, and contingent experience. The memory palace, like the larger palimpsest of habitations, pathways, and meanings that form our public buildings and cities, was designed as a walking space of inscription and recall, of erasure and forgetting, … and of reinscription. And the workings of the palace – the method of loci– bear centrally upon the logics of locative media.
A mnemonic technique based on spatial and visual logics, the memory palace took root in ancient Greece and Rome, its use described by rhetoricians from the latter including both Cicero and Quintilian. Tasked with memorizing a complex oration, the practitioner of the method of loci essentially maps memory triggers into an imaginary architectural space. By rehearsing the passage through space (or remembering an already familiar space) and mentally adorning the rooms with distinctive visual associations – ‘markers’ – not only can the user render the ‘palace’ into an organized repository of changeable memory cues, but she can easily change her routing and access different ‘rooms’ and their memory cues more or less on demand. This last point underscores the added value of the spatial, architectural model. Just as with our own house, should we suddenly want something from the hall or the kitchen, we know how to get there directly from any location rather than first taking a preordained route. At least for the ancients, the main mnemonic alternatives – learning by rote and rhyme – erred on the side of reinforcing a preordained, linear, route. This difference has implication, not the least for the notion of narrative.
Much as with Culler’s tourist, ‘markers’ do the heavy lifting of “constituting a sight as a sight, giving information about it, representing it, making it recognizable.” [8] In Culler’s case, the markers are part of a broadly cultural repertoire of knowledge; in the memory palace, they exist at the whim of the user. This assemblage of associative logics, mutable and assignable meanings, and an architectural conceit for the structure of memory required the user to activate the space, to perform it by (mentally) wandering through it. And despite the many paths that one might take through the palace (as opposed to the fixed structure of rote and rhyme), that activation seems as though it was narratively charged in a double sense: one could go from one room to another hewing to a well-rehearsed path with the familiar fixity of textual structure and events retold; or one could search the house with purpose, encountering the markers in a particular motivated sequence, and in the process, render them narratively coherent. Narrative, in this latter sense, turns on a purely subjective unfolding of motivated and sequenced revelations, on a process of assemblage and an experience of coherence rather than the consumption of a pre-structured text.
Culler’s tourist, the pilgrim on the Via Dolorosa, Ginzburg’s hunter and diviner, and the individual meandering through the memory palace, all share navigational and associational strategies that transform and inform their worlds through the use of markers. But Culler’s tourist and the pilgrim inhabit an existing and declared narrative structure, and wend their way through it with the help of public markers; whereas Ginzburg’s hunter and diviner, armed with specialized knowledge and ways of seeing, construct narratives that only they and others equally skilled can decipher. The habitué of the memory palace, by contrast, wandering among her own associations, draws upon the same basic techniques as the others, but for a completely subjective narrative end whether by way of a well-rehearsed pathway or an ad hoc discovery experience. In each case, acts of indication are selectively legible, whether public, limited by skill-sets and interests, or purely subjective, working their semiotic magic for those in the know and either revealing the contours of a pre-existing narrative or dropping the breadcrumbs that will construct a new one.
Before turning to the implications of these precedents for the augmentation of reality in urban spaces, it is worth noting that recent scientific findings offer a resonant neurological model for how we navigate in the spatial world. From initial research in the early 1970s to the award of a Nobel Prize in 2014, work on grid cells and place cells has offered an increasingly convincing model for how rats – and by implication, humans – navigate. [9] The brain’s grid cells take the form of a stable hexagonal lattice, akin to the fixed floor plan of the memory palace; it provides an abstract overlay for both unknown spaces and familiar ones, one plan fitting all. Place cells mark variable positions or spatial events on the grid, akin to the vivid memory triggers that the user deploys in the memory palace: markers. They help to transform the abstract grid into something specific, marking unique spots. While this discovery quickly led to the somewhat controversial speculation that place cells encode cognitive representations of Euclidian space in the brain, their role in dynamically computing self-position and enabling spatial navigation based on a grid and continuously updated place ‘events’ seems widely accepted. The combination of the fixed, abstract map and the ‘programmable’ and dynamic place cell echo the logics of the memory palace, and offers an intriguing analogy to augmented reality as a spatial practice.
A Brief Excursus on Augmented Reality
Thus far, we have considered a few moments in the history of ‘markers,’ acts of indication that occur by conjoining space, memory, and meaning, whether declaratively public, selectively venatic, or just plain personal. We have seen various deployments of the method of loci as a “Generative friction between different formats,” to use Galloway’s words, offering a means for the semiotic augmentation of the world. [10] And we have seen that more often than not, an expanded sense of narrative is implicated in this process: narrative as a public pathway marked by established signs (a framing strategy that enables Culler’s ubiquitous semiotic mechanisms that give social orders their coherence), but also narrative as an experiential process activated by certain agents who are able to decipher signs into coherence.
This double sense of narrative, that is, narratives as pre-existing and collectively understood texts and narratives as coherent aggregations of experience and ways of seeing, is important. At one extreme, markers (whether Culler’s or the memory palace’s ‘vivid images’) punctuate a spatially embedded public narrative. As suggested both by Culler and the Via Dolorosa example, markers overlay the streets and doorways of the material world, providing a palimpsest of associations at once residual, well-rehearsed, and newly acquired, but always standing in relation to a historically accreted – and publically endorsed and indeed, declared – narrative. At the other extreme, markers enable a process of meaning-making. Hunters and diviners use their special venatic knowledge to decipher and piece together various signs into an unfolding narrative that only they or others with equivalent skills can discern. Practitioners of the memory arts could have it both ways, enjoying the affordances of their mental constructions to embed their experiences in markers and spaces that, as the situation requires, either punctuated a well-rehearsed personal narrative, or that might be discovered by ruminating through the palace and linking findings together into a purpose-driven coherence (exactly the point of Camillo’s Theater of Memory, which was more of a memory machine than memory space [11]). Thinking about markers as both punctuation and process offers an entry point to thinking about shared spaces, whether architectural or urban, as repositories of memory, narrative, and meaning.
These markers, often in the form of vivid images, punctuate the urban grid-like place cells, giving a relatively abstract map or plan both specificity and meaning. Of course, the map is a site-specific representation, not a universal overlay, so the grid and place cell analogy might seem overdrawn. But at least since Mercator’s day, our maps have been composed of two elements: a representational order (say, the visual contours of a coastal area) and an abstract overlay of three-dimensional Cartesian coordinates (longitude, latitude, elevation) … a grid. Our latest class of ‘markers’ in the form of augmented reality makes use of this latter grid, even as it guides us through the representational level of the map.
Michel de Certeau’s Practice of Everyday Life offers an important and well-received way of articulating (among other things) these quotidian encounters. [12] De Certeau charts a number of relevant distinctions, for example between the map (an abstract rendering of an area) and the tour (its inscription with the fullness of experience and meaning), or between place (“an instantaneous configuration of positions,” “an indication of stability” [13]) and space (“the effect produced by the operations that orient it, situate it, temporalize it, and make it function in a polyvalent unity of conflictual programs” [14]). Put more eloquently, De Certeau says that “space is a practiced place.” [15] Similar to the method of loci’s transformation of location through the addition of associations, ‘place,’ once situated, historicized, and laden with meaning, emerges as ‘space’; and the ‘map,’ once interpreted, contextualized, and larded with meaning, emerges as the ‘tour.’ Familiar places and passages through city can be inscribed with associations – either in a manner that emulates the memory palace (‘wandering’ through an entirely mental representation of the city, our associations trigger the things we earlier planted for subsequent recall), or in a lived and intersubjective way (walking through the city, sights, and locations trigger memories of historical encounters). The mechanics and specificities of the ‘added value’ that transforms the ‘map’ and ‘place’ to the ‘tour’ and ‘space’ constitute the domain of augmentation, and in its instantiation, interface.
In order to consider the project of augmented reality and where it fits with the traditions briefly sketched to this point, we can turn to a site that is itself also an argument. The site is the city, and the argument resides in the nature of cities as palimpsests, material constellations of time testifying to their multi-layered pasts and the long-term processes of their modification, overwriting, and neglect. Urban spaces are loaded with signification bearing the iconic markers of the past’s dominant and declared narratives (Culler’s tourist sites and the Via Dolorosa); providing hidden and yet-to-be-deciphered signs available to those of a venatic bent; and serving as fodder for countless individual experiences, associations, and memories. In each case, the added semiotic value is contingent on some technique or technology and the user’s relationship to it. While it might seem a stretch to consider these overlays as interfaces, they certainly produce ‘a generative friction’ between different experiential layers. This history of endeavors sets the stage for considering contemporary practices of annotation and augmentation of everyday interactions with the spatial environment clustered under the fast-changing rubric of ‘augmented reality,’ currently a subset of virtual reality.
For purposes of disambiguation, I will use ‘augmented reality’ and ‘virtual reality’ to indicate broad conceptual projects, and their capitalized initials (AR and VR) to refer to currently available consumer technologies that operate in these spaces. As I write this, virtual reality refers to a computer-generated emulation of a ‘real’ world, often in the form of a space that the user can interact with and explore. It can use visual, haptic, acoustic, and even olfactory cues to evoke a world that is realistic … or not. Out of this broad definition, several branded technologies have emerged and fought for a share of the consumer market. Yet even here, there is plenty of room for ambiguity, as demonstrated by the contenders for the VR title: at one end of the spectrum, the simplicity of VR as 360 video has led to widespread use (the New York Times has a daily 360 feature, and Youtube has a 360 channel); and at the other end, VR systems that enable more robust interaction with their worlds (LIDAR, Kinect, and photogrammetry-based real time capture systems) are still expensive and can be a bit fussy technologically-speaking. Although these various VR products claim the mantle of virtual reality, they point in radically different directions, with different technologies, capacities, aesthetics and even ethical considerations. VR in the form of 360 video is the conceptual descendant of Robert Barker’s 1787 panorama – a fixed visual asset in which the viewer can look around, and possibly trigger a few hot-spots. VR in the form of real time capture systems, by contrast, enables interaction, with visual assets effectively being generated on the fly as the viewer moves around.
This all matters because, technically speaking, augmented reality is a subset of virtual reality (‘…a computer-generated emulation…’), but with an important twist. Whereas VR, as we currently construct it, requires immersion in a computer-generated world, augmented reality in the form of AR provides a virtual overlay on the world. That is, we see simulated and geo-located data or characters or images or hear sounds as well as sharing the experiences of the larger world. We can still easily interact with the world (and, depending on the system, with simulated artifacts), and AR lends itself to broader participation in the sense of allowing people to contribute their own virtual assets to the system (something much more difficult with VR). Technologically, AR visual headset systems like Magic Leap and Microsoft’s Hololens (marketed under the confusing rubric of Mixed Reality) still have a way to go, although projectors, mobile phones, tablets, and even audio headsets can also serve as ‘portals’ enabling users to see augmented overlays on the world. With VR, we leave the real world behind in order to enter a closed, simulated world; whereas with AR, we append a simulated layer onto the real world, and interact with both.
AR’s informational overlay is tagged to particular places, and at the moment, three different systems may be used to link the real and the virtual (in addition to projected overlays on the world). First, fiduciary markers are the most basic, consisting of graphically coded tags that are physically attached to the object for which an overlay is sought. The receiving device’s recognition of the code triggers the appearance of the data layer in its screen, treating the information or image as spatially substantial (one may, in effect, walk around a virtual object as if physically exists). A second and more pervasive application makes use of the Cartesian grid in the form of geo-positioned data and GPS (Global Positioning System), a compass and an accelerometer. Wikitude, for example, calculates users’ positions by using these elements, linking them together with the Wikitudedata set to provide location-based information, imagery, etc., just as in the fiduciary case. [16] Finally, natural feature tracking systems represent a fast-emerging image technology that assigns data to location by making visual correlations between physical places (i.e. ‘recognizing’ them) and the information to be appended. An image-recognition system, it requires the user to position sights within a viewfinder, which are then processed to find any correlations with the stored database. When a correlation is found, as in the previous cases, an information overlay appears. The system’s search for unique identity points is a form of machine vision, except that in this case, the user in the physical world attempts to correlate real and virtual data in order to optimize the chances of a virtual graphic overlay. For the moment, natural feature tracking’s intensive processing demands have limited its use to powerful handhelds and relatively iconic locations, but among industry insiders, it seems to be the application of choice.
Augmented reality applications such as Layar and Wikitude, even though they use digital compass and not natural feature tracking, can overlay dynamically responsive information in the screen of the handheld; point out locations of likely interest (based on aggregated user information); and display them as overlays in the device’s screen, linking those locations to web-based information repositories. [17] Applications range from entertainment (Pokémon Go and Ingress, which will be discussed shortly), to data overlays regarding urban infrastructure (water, sewage and electrical conduits, etc.), to heritage and tourism related information, to, of course, location-specific advertising, marketing, and much more. [18]
Public Memory
How does one ‘perform’ public memory, or what generally passes as history? The modern era has tended to rely upon prosthetic devices like books (the very instruments that marked the end of the memory arts) along with the plaques, commemorations, tours, etc. that Culler describes as among the ‘semiotic mechanisms that give social orders their coherence.’ These elements and others of their kind punctuate historical narratives, marking key moments and places, and helping to extend the broad contours of selected pasts into our present. [19] They aren’t the nuanced ‘stuff’ of professional historiography, so much as a visible residue that keeps that ‘stuff’ in the public’s imaginary. Issues such as media specificity obviously complicate this (written narratives relay different types of information from, say, statuary [20]); and the present is always complicit in the recognition of a historical past. These and other qualifiers notwithstanding, the point is that the nature of the markers is keeping pace with the culture’s larger technological and participatory shifts.
As I write this essay, the city of Montréal, as part of the celebration of its 375th anniversary, offers several striking examples of augmented reality of the declarative, public variety, with markers inscribed on the surfaces of city and its buildings. These markers align conceptually with the earlier discussion of both Culler’s tourist spots and the Via Dolorosa example, with the augmentation of space helping to punctuate an already known narrative, and doing so in a highly visible manner. Cité Mémoire, billed as an ‘urban multimedia experience,’ is an ambitious installation mounted by Michel Lemieux and Victor Pilon in collaboration with playwright Michel Marc Bouchard. [21] The cross-media project is made up of large-scale light projections on the façades of old Montréal’s buildings and trees, sound tracks via the smartphone, a website, and street signage. The project’s promotional text summarizes the intent:
The Tableaux presented in Cité Mémoire cover a range of milestones in Montréal’s history, well-known (or not so well-known) characters, or sometimes a local slice of life. Narrative or impressionist, touching or amusing, they express our values of coexistence, innovation, tolerance and generosity that are the foundations of our city. [22]
Although daytime visitors to old Montréal will likely notice the apparatus of projectors, stenciled-on-the-sidewalk viewing positions, and the initiative’s promotional materials, the project itself is best viewed in the evening. Historical references are writ large, significantly surpassing the scale of any nearby illuminated advertising, and beckoning passersby’s attention with visually alluring enactments of moments in the city’s past. By literally overlaying the city’s surfaces, visual references to its many and complex histories are rendered emphatically public, even though hardware in the form of a smartphone is required to hear the location-specific sound tracks. Conceived as a multimedia event, Cité Mémoireobviously requires both image and sound for full effect, even if the acoustic dimension is less overtly public than the visual. Nevertheless, the augmentation of reality is visually powerful, and sufficient to punctuate the underlying narratives.
As part of the same 375th anniversary celebration, Montreal’s Notre-Dame Basilica offers a variation worth noting. AURA, an immersive experience designed by the Moment Factory team (including Sakchin Bessette, Dominic Audet, and Éric Fournier), uses projection mapping and orchestral accompaniment to animate the Basilica’s statuary and paintings, and ultimately to construct a loose narrative about history and spirituality. [23] Apart from the impressive visual technology deployed, it differs from Cité Mémoirein two key ways: the soundtrack is as unmissable as the images – no special technologies are needed since the sound is ambient; and the notion of ‘public’ is constrained by architectural enclosure and an admission price, that is, it is framed as paid entertainment. Within the confines of a semi-public space, however, the augmentation process is evident for all visitors, and the project feels consistent with the earlier references to gothic cathedrals, where everything from statuary to performance serve to punctuate a basic cultural narrative. ‘Public’ is a relative term, and sites from the Via Dolorosa to the local cathedral are relatively open, although obviously not universally so. In this case, the markers usually associated with the space are themselves augmented and transformed from their everyday taken-for-grantedness; re-charged, they assert themselves as markers and do so with affective power.
Hunters and Gatherers
As Ginzburg shows with his hunters and diviners, markers – even if publically sited – are sometimes only visible to those with special knowledge or who are in search of a particular goal. We might extend this filtration process by also including technology: a new generation of augmentation practices are only visible to those with the appropriate device, software, and know-how. Roundware, Halsey Burgund’s open source “audio augmented reality for art, education and documentary,” enables the collection, storage, organization, and representation of layered and mixed-on-the-fly audio content. [24] Users can tag audio content, geo-locating it where they will. The center of the Pentagon, the heart of the US Department of Defense? No problem: all one needs are the geographical coordinates. The trick, of course, is that someone in the Pentagon needs to know that the tag exists, and have the proper software installed on their mobile audio device. A more accessible application of Roundware is re~verse, a continuous layer of location-sensitive music with recorded voices of poets who have visited Harvard and given readings of their work. [25] This audio augmentation project can be experienced on location in Harvard Yard if the user knows that it exists and has the requisite software, encouraging a process of discovery as one poet leads to another, and the story of Harvard’s involvement with the art unfolds. But just as in the hypothetical Pentagon example, it is also easy to miss if one lacks the essential discovery components.
The process of augmentation, of marking, in spaces that are technically public but in ways that are only selectively accessible, takes many other forms. Yellow Arrow(2004-2006), a psychogeography-inspired art project developed by Christopher Allen, Brian House, and Jesse Shapins, made use of coded yellow stickers and mobile phones. [26] Yellow Arrow’s original website described the project’s operations as follows:
Participants place uniquely-coded Yellow Arrow stickers to draw attention to different locations and objects—whether a back-alley mural, a favorite dive bar, or a new perspective on a classic landmark. By sending an SMS from a mobile phone to the Yellow Arrow phone number beginning with the arrow’s unique code, Yellow Arrow authors essentially save a thought on the spot where they place their sticker. Messages range from short poetic fragments to personal stories to game-like prompts to action. When another person encounters the Yellow Arrow, he or she sends its code to the Yellow Arrow number and receives the message on their mobile phone. They can then reply to send a message to the author. The website yellowarrow.org extends this location-based exchange, by allowing participants to annotate their arrows with photos and maps in the online gallery of Yellow Arrows placed throughout the world. By collecting and sharing places of personal significance, this public collaboration expresses the unique characteristics, personal histories, and hidden secrets that live within our everyday spaces.[27]
Those ‘in-the-know’ kept an eye out for the discrete stickers and the stories and interactions linked to them. And once ‘in-the-know,’ followers could track down other arrows through the project’s website. But to most of the passing public, the yellow tags were invisible: simply part of the urban mix of sticker graffiti and street art … rather than gateways to memories, stories, and perspective-transforming insights.
Both Yellow Arrow and Roundwareare audio-centric examples of augmentation, one with a visible interface, and the other more discrete in its visible markings, even if both work simultaneously to address and construct communities of knowledge. Several of the most popular – and literal – manifestations of venatic augmentation systems take game form: Niantic’sIngress(2012+) and its successor, Pokémon Go(2016+). Niantic, a Google spinoff, allegedly created Ingress as a proof of concept for other AR games built on Google Maps’ data. Targeted to a niche market of gamers, widely varying user reports put player numbers at between five hundred thousand and seven million, with most settling for just shy of a million. [28]Gameplay consisted of searching out and capturing ‘portals’ at places of cultural significance, such as monuments, landmarks, and public art installations, and linking them to create virtual ‘control fields’ over geographical areas. Tied to a science fiction back story, the game app generated augmented maps of the play area, and tracked progress as two teams vied to ‘save humanity’ in this open-ended discovery-and-conquest narrative.
Promotion of the game was largely viral, and apart from the odd behavior of players, little in the physical world would provide clues as to the markers contested by the players and the virtual reality overlaying the everyday world. Curiously, one of the reasons to locate gameplay at monuments, etc., was to encourage players to explore their cities; alas, augmentation seems to have been too effective, for while it generated traffic, the fictional overlay seems to have predominated. The game’s business model also sought to populate game sponsors’ commercial locations through a kind of location-based advertising, but this too seems to have yielded mixed results. [29]
Ingress’s modest success aside, its mapping data and points of interest were used to select the locations for Niantic’s subsequent game, Pokémon Go, released in July 2016. [30] Pokémon Goutilizes the player’s mobile device’s GPS to locate, capture, battle, and train virtual creatures which appear on the screen as if they were at the same real-world location as the player. Niantic, by this point independent from Google, had learned valuable lessons. By August 2017, Pokémon Go’s software was downloaded 750 million times worldwide, with a monthly active user base of around 65 million and 2016 annual revenue of close to one billion USD. In addition to functioning as a hunter-gatherer game, Pokémon Gowas also about marketing, and it succeeded here as well, with some 35,000 sponsored locations and 500 million visits to those locations.
Even more literally than with re~verseand Yellow Arrow’s various deployments, users of games like Ingress and Pokémon Go search for markers that are visible and meaningful only to them – that is the point of the games (commercial drivers aside!). Of course, as Ginzburg reminds us, the world abounds in signs legible only to select populations on the basis of disposition and ability; but in these AR cases, an additional precondition to legibility is technological. The technological filter – the requirement of having the hardware, software, and knowledge necessary to access the markers embedded in a layer of augmentation – characterizes the vast majority of AR applications. WhetherLayar-based real estate applications, or tourism and heritage apps, or industrial uses, the user must first know about an order of signs, and then enable the technology to access it, before the augmented experience. Truly public augmentations of reality, such as those associated with Cité Mémoire, are exceptional insomuch as their public status requires a particular order of technology (a low threshold and unmissable technology such as projection), and because of that ‘unmissability’ generally requires political consent. Like the plaques, statues, and memorials that have historically punctuated public memory, this new generation of augmentation is also bound by the collective order. But of course, this is a half-truth, since commercial billboards and illuminated advertising vie for attention with their somewhat duller public brethren. Projection mapping is of course being drawn upon by advertisers, just as Pokémon Go’s choice of locations is a commercially driven source of revenues. But for the moment, most of these commercial applications stand as one-off assertions of brand or product, not augmentation in the sense of location-specific punctuation with larger narrative potential. And fortunately, as these commercial applications inevitably unfold, so too are counter-deployments of the same projection mapping technologies for purposes ranging from protest, to graffiti, to art. [31]
A Trip Down Memory Lane…
The memory palace figured prominently in the set up for this discussion, offering an early and well-practiced precedent for the augmentation of space and ambulatory access to the information stored there. While useful as a model and resonant with recent discoveries about spatial memory and navigation (grid and place cells), the memory palace is ultimately mired in subjectivity, a figment of the imagination. Most current discussions about AR, by contrast, tend to focus on expanding the claim on public space and making the discovery process bound up in the venatic tradition more findable, compelling, and shareable. Not surprisingly, those discussions are generally commercially driven. But is there a place for private AR, for literalizing the metaphor of the memory palace and extending our individual markers to the public spaces we inhabit and traverse? Such a move would augment space for our eyes only, serving as a personal archive.
Conditions certainly seem right for such a development. Most digital cameras include options to capture geo-locative and temporal information, along with image data. Thanks to the ubiquitous access enabled by cloud storage systems, it is but a small step to use these data to append images to the places in which they were taken, and to organize them temporally as well. Apps such as EthnoAlly, a mobile phone-based digital research tool, enable users to record audio-visual, geolocative, and textual materials as a way to “coordinate multiple acts of serendipitous ethnography.” [32]The resulting diary-like photographs, videos, audio-recordings, and texts – all geolocated – could easily be re-located to their points of origin. At an extreme, the sousveillance initiatives associated with Steve Mann’s ‘lifelogging’ and Justin Kan’s ‘lifecasting’ yield a continuous flow of personal information that could, should the user desire, be appended to the locations from which they came. Now that storage and sites of access are disaggregated, there is nothing stopping those who wish to transform the world into a personal memory palace by attaching images to the places they inhabit. Indeed, it might be argued that, given that aggregation and near ubiquity of access, something like personally augmented reality is already here, if accessing our historical audio-visual material on-the-spot is its measure. But ubiquity of access and geolocated data are different if overlapping configurations, and geolocated information prompts the memory, rather than, as with ubiquitous access, memory prompting the download.
There is something to be said for augmenting spaces with the markers of our individual pasts. Like the examples of public memory that we saw earlier, such augmentations can punctuate an oft-told narrative – the user’s moves up (or down) the social ladder, from one neighborhood or domicile to another; or through the educational system. And like the earlier venatic examples, augmentation can also offer a process of narrative discovery, as the user finds new associations and coherence through newly sequenced encounters with their own markers. Like the classic memory palace, coherence can take either narrative form in this speculative use of AR.
Conclusion
Let’s return to Jonathan Culler on tourism. Culler says:
A semiotic perspective advances the study of tourism by preventing one from thinking of signs and sign relations as corruptions of what ought to be a direct experience of reality and thus of saving one from the simplistic fulminations against tourists and tourism that are symptoms of the touristic system rather than pertinent analyses. Tourism, in turn, enriches semiotics in its demonstration that salient features of the social and natural world are articulated by what Percy calls ‘symbolic complexes’ and its revelation of the modern quest for experience as a quest for an experience of signs. Its illustration of the structural incompleteness of experience, its dependency on markers, helps us understand something of the nature of semiotic structures. [33]
Culler’s words, although written for a different purpose, resonate with the semiotic overlays on the world that we call augmented reality. They serve as a reminder, in part, that existing analytic frameworks (in his case, semiotics) remain relevant; as a reminder of the reciprocity between theory and the practices of everyday life; and as a reminder of an underlying condition – the structural incompleteness of experience, the quest for an experience of signs – that continues to shape human activity. Our habit of framing reality – of categorizing, articulating, reading and inscribing it with meaning – seems, if we follow Ginzburg’s argument, a defining trait of our species. It accounts for our success as hunters and gatherers, and for the prominence of narration as an organizing principle.
The forms of AR that populate the floors of electronics trade shows are laden with as-yet unfulfilled promise. Those developments are largely outside the prevue of this essay, even though they constitute perhaps the most obvious sites of technological interface. Instead, I’ve tried to address the underlying drivers for the augmentation of reality, and with them, some sense of their deeper rooting in our cultural practice, our quest for signs, and our reliance on interfaces for semiotic coherence and orientation. If virtual reality has an experiential reference, it is the dream, complete and compelling immersion in an ‘other’ reality. And that so-proximate yet technologically unachievable state, I suspect, will ultimately be VR’s undoing. But the case for augmented reality is different. Whether public, discovery-based, or even personal, augmentation’s inherently dialogic character requires reality for its work. And its work turns on the same semiotic reciprocities that humans have engaged in over the long haul. Technologies will come and go, but humans will continue to mark the world, read it, and find coherence – activities inherent in augmented reality.
Author Biography
William Uricchio is professor of Comparative Media Studies at MIT (US) and professor emeritus at Utrecht University (Netherlands). Founder and principal investigator of MIT’s Open Documentary Lab, he explores the frontiers of interactive, immersive, locative, and participatory reality-based storytelling. His scholarly research considers the interplay of media technologies and cultural practices in relation to representation, knowledge and publics. A specialist in old media when they were new, he explores such things as early 19th C conjunctures between photography and telegraphy; the place of telephony in the development of television at the other end of the 19th C; and the work of algorithms in our contemporary cultural lives. William has held professorial appointments in Sweden (Stockholm), Germany (FU Berlin, Marburg), Denmark (national DREAM professor) and China (China University of Science & Technology), and has received Guggenheim, Humboldt and Fulbright awards, the Berlin Prize, and is a Mercator Fellow.
Notes and References
[1] William Uricchio, “The Algorithmic Turn: Photosynth, Augmented Reality and the Changing Implications of the Image,” Visual Studies 26, no. 1 (2011): 25-35.
[2] Jonathan Culler, Framing the Sign: Criticism and Its Institutions (Norman, OK: University of Oklahoma Press, 1990), 5.
[3] Alexander Galloway, The Interface Effect (New York, NY: Polity Press, 2012), 30.
[4] Carlo Ginzburg, Clues: Roots of an Evidential Paradigm (Baltimore, MD: Johns Hopkins University Press, 1989), 103.
[5] Ibid.
[6] Ibid.
[7] Ibid.
[8]Culler, Framing the Sign,159
[9] May-Britt Moser, Edvard Moser, and John O’Keefe were awarded the Nobel Prize in Physiology or Medicine in 2014 for their work on grid cells.
[10] Galloway, The Interface Effect, 30.
[11] I have elsewhere pursued the differences between the memory palace and Camillo’s 15th Century theater of memory and its related though distinct project. Camillo’s theater offers the possibility of a common set of associations that any passerby may use, rather than the inherent in the memory palace. These associations are systematized, organized in a manner similar to indexes and charts in how they structure information. The associations may be vivid and graphic (in the city, just as in theater of memory, as well as in the memory palace), but their significance is intersubjective. We might look for parallels between the operations of Camillo’s theater and some of today’s technologies, such as location-based technologies, augmented reality, geo-tagging systems, etc. A battery of new possibilities has changed our access to the city. The near-ubiquity of powerful smartphones and handhelds equipped with these and other applications has, in a sense, given the urban spectator fingertip-access to a vast dataset. Like Camillo’s theater, an encyclopedic array of data is at the user’s disposal, allowing the flâneur-spectator to connect information, and both activate and create memories. This scenario entails much more than simply walking through place (in De Certeau’s sense); it turns on the continual creation of space through the accessing and activation of data. It simultaneously historicizes, narrativizes, and personalizes the encounters between the subject and the city, generating a rich experience from the structure of associations. See William Uricchio, “A Palimpsests of Place and Past,” Performance Research 17, no. 3 (2012): 45-49.
[12] Michel de Certeau, The Practice of Everyday Life(Berkeley, CA: University of California Press, 1984).
[13] Ibid., 117.
[14] Ibid.
[15] Ibid.
[16] Wikitude, “SDK∞Endless AR Possibilities,” accessed July 4, 2018, https://www.wikitude.com/.
[17] Layar, “Quick and Easy Self-Service Augmented Reality,” accessed July 4, 2018, https://www.layar.com/features/; Wikitude, “SDK∞Endless AR Possibilities.”
[18] The Pokemon Company, “Pokémon GO: Get up and Go,” accessed July 4, 2018, http://www.pokemongo.com/; Ingress, “The world around you is not what it seems,” accessed July 5, 2018, https://www.ingress.com/.
[19] William Uricchio and Roberta E. Pearson, Reframing Culture: The Case of the Vitagraph Quality Films(Princeton: Princeton University Press, 1993).
[20] Seymour Chatman, Coming to Terms: The Rhetoric of Narrative in Fiction and Film(Ithaca, NY: Cornell University Press, 1990).
[21] Montréal En Histoires, “Cité Mémoire,” accessed July 5, 2018, http://www.montrealenhistoires.com/en/cite-memoire/.
[22] Ibid.
[23] Aura Basilica Montreal, “AURA: A Luminous Experience in the Heart of the Basilica,” accessed July 5, 2018, http://www.aurabasiliquemontreal.com/en/.
[24] Roundware, “Contributory Audio Augmented Reality,” accessed May 5, 2019, http://roundware.org.
[25] Halsey Burgund, “re~verse,” accessed July 5, 2018, https://halseyburgund.com/projects/reverse.html. This project was made by Halsey Burgund in collaboration with Harvard’s metaLAB and the Woodberry Poetry Room. Future plans for Roundwareinclude the ability to geo-tag photos, videos, and text.
[26]Brian House, Christopher Allen and Jesse Shapins, “Yellow Arrow,” accessed July 5, 2018, https://brianhouse.net/works/yellow_arrow/.
[27] Ibid. For a thorough discussion of both Yellow Arrow and Roundware, see Sue Ding, “Re-Enchanting Spaces: Location-Based Media, Participatory Documentary, and Augmented Reality” (master’s thesis, Massachusetts Institute of Technology, 2017).
[28] Louise Beltzung, “The King of Augmented Reality Street Fighting,” Vice, January 8, 2015, https://motherboard.vice.com/en_us/article/z4mgzy/the-king-of-augmented-street-fighting-the-worlds-best-ingress-player;Sam Rutherford, “Meet Endgame: Proving Ground – Google’s AR Mobile Game,” Tom’s Guide, March 16, 2015, https://www.tomsguide.com/us/endgame-proving-grounds-beta,news-20650.html.
[29] Some controversy surrounds Ingress, as suggested by the title of a 2012 review: “Google Accused Of ‘Blatantly’ Ripping Off Grey Area Games’ Shadow Cities,” see Chris Priestman, Archive, November 27, 2012, https://archive.is/20121129084454/http://indiestatik.com/2012/11/27/google-accused-of-blatantly-ripping-off-grey-area-games-shadow-cities/.
[30] For a critical genealogy of Ingress and Pokémon Go, see Ian Bogost, “The Tragedy of Pokémon Go: What it Takes for Good Ideas to Attract Money,” The Atlantic, July 11, 2016, https://www.theatlantic.com/technology/archive/2016/07/the-tragedy-of-pokemon-go/490793/
[31] Urban Projection, “Street Projection,” accessed July 5, 2018, https://www.urbanprojections.com/street-projection.
[32] AppAdvice, “EthnoAlly by Pablo Fernandez Muga,” accessed July 5, 2018, https://appadvice.com/app/ethnoally/1104947792.
[33]Culler, Framing the Sign, 9.
Bibliography
AppAdvice. “EthnoAlly by Pablo Fernandez Muga.” Accessed July 5, 2018. https://appadvice.com/app/ethnoally/1104947792.
Aura Basilica Montreal. “AURA: A Luminous Experience in the Heart of the Basilica.” Accessed July 5, 2018. http://www.aurabasiliquemontreal.com/en/.
Beltzung, Louise. “The King of Augmented Reality Street Fighting.” Vice, January 8, 2015. https://motherboard.vice.com/en_us/article/z4mgzy/the-king-of-augmented-street-fighting-the-worlds-best-ingress-player.
Bogost, Ian. “The Tragedy of Pokémon Go: What it Takes for Good Ideas to Attract Money.” The Atlantic, July 11, 2016. https://www.theatlantic.com/technology/archive/2016/07/the-tragedy-of-pokemon-go/490793/.
Burgund, Halsey. “re~verse.” Accessed July 5, 2018. https://halseyburgund.com/projects/reverse.html.
Chatman, Seymour. Coming to Terms: The Rhetoric of Narrative in Fiction and Film. Ithaca, NY: Cornell University Press, 1990.
Culler, Jonathan. Framing the Sign: Criticism and Its Institutions. Norman, OK: University of Oklahoma Press, 1990.
De Certeau, Michel. The Practice of Everyday Life. Berkeley, CA: University of California Press, 1984.
Ding, Sue. “Re-Enchanting Spaces: Location-Based Media, Participatory Documentary, and Augmented Reality.” Master’s thesis, Massachusetts Institute of Technology, 2017.
Galloway, Alexander. The Interface Effect. New York, NY: Polity Press, 2012.
Ginzburg, Carlo. Clues: Roots of an Evidential Paradigm. Baltimore, MD: Johns Hopkins University Press, 1989.
House, Brian, Christopher Allen, and Jesse Shapins. “Yellow Arrow.” Accessed July 5, 2018. https://brianhouse.net/works/yellow_arrow/.
Ingress. “The world around you is not what it seems.” Accessed July 5, 2018. https://www.ingress.com/.
Layar. “Quick and Easy Self-Service Augmented Reality.” Accessed July 4, 2018. https://www.layar.com/features/.
Montréal En Histoires. “Cité Mémoire.” Accessed July 5, 2018. http://www.montrealenhistoires.com/en/cite-memoire/.
Priestman, Chris. “Google Accused Of ‘Blatantly’ Ripping Off Grey Area Games’ Shadow Cities.” Archive, November 29, 2012. https://archive.is/20121129084454/http://indiestatik.com/2012/11/27/google
-accused-of-blatantly-ripping-off-grey-area-games-shadow-cities/.
Roundware. “Contributory Audio Augmented Reality.” Accessed May 5, 2019. http://roundware.org.
Rutherford, Sam. “Meet Endgame: Proving Ground – Google’s AR Mobile Game.” Tom’s Guide, March 16, 2015. https://www.tomsguide.com/us/endgame-proving-grounds-beta,news-20650.html.
The Pokemon Company. “Pokémon GO: Get up and Go.” Accessed July 4, 2018. http://www.pokemongo.com/.
Urban Projection. “Street Projection.” Accessed July 5, 2018. https://www.urbanprojections.com/street-projection.
Uricchio, William. “The Algorithmic Turn: Photosynth, Augmented Reality and the Changing Implications of the Image.” Visual Studies 26, no. 1 (2011): 25-35.
Uricchio, William. “A Palimpsests of Place and Past.” Performance Research 17, no. 3 (2012): 45-49.
Uricchio, William, and Roberta E. Pearson. Reframing Culture: The Case of the Vitagraph Quality Films.Princeton: Princeton University Press, 1993.
Wikitude. “SDK∞Endless AR Possibilities.” Accessed July 4, 2018. https://www.wikitude.com/.