-- online since 1993
ISSN NO : 1071 - 4391 The MIT Press
Lens: The Practice and Poetics of Writing in Immersive VR
v o l 1 4
i s s u e

Lens: The Practice and Poetics of Writing in Immersive VR
(A Case Study with Maquette)

Click here to download pdf version.

by John Cayley

with Dmitri Lemmerman
Java Developer

writing in programmable media; digital poetics, spatial poetics, immersive VR virtual reality, Cave writing, poetics, theory, textual animation, new writing practices, electronic poetics, writing on complex surfaces, theories of writing and textuality, the phenomenology of text in 3D space

Faced with the prospect of collaborating on writing projects sited in Brown University’s four-wall VR Cave, at least two major issues confront the practitioner. One has socio-political and economic implications: How can such a recondite and exclusive process of artistic making address audiences or generate a socially engaged practice? The other is formal, rhetorical and, I would argue, crucial to culture at the present moment: What is, what will be, the phenomenology and aesthetics of text in 3-D space? These questions are large and daunting.

In the following essay and report, it is primarily the second of the two questions that will be addressed by way of a case study involving two works-in-progress that Cayley has initiated in Brown’s Cave with the development assistance of Dmitri Lemmerman. Both the form and content of these pieces emerged from an experimental approach to the Cave's media. Cayley came to the Cave with ideas that might have led to the transposition of suitable existing writing in programmable media to the Cave. Typically, his work targets the 2-D screen with its inscribed surface and illusionistic depths, but Cayley’s resolutely textual approach had previously engaged topology as concept rather than as (illusionistic) representation. To take the conceptual topology of such work and translate it to immersive VR was an obvious first step, but what would this mean and how would it be read, as writing?

None of those working on the project were prepared for the discoveries that were made in the process of transposition, concerning, to give one example, the ability of text to define and delineate 3-D spatial structure. This case study outlines some of our discoveries and implications for the broader questions. In a sense, the fact of the discoveries, answering formal and rhetorical problems, begins to address, if not necessarily justify or explain, the sociological issues underpinning a novel practice of poetic art in programmable media.



Figure 1 © John Cayley

Figure 1
Copyright © John Cayley

The narrative and much of the expository part of this essay will be founded on my report of graphic and literary compositional activities in Brown University’s virtual reality Cave. The requisite institutional relationships — for which, I should acknowledge, I continue to be extremely grateful — tend to enforce the bracketing of direct, and particularly confrontational, engagement with socio-political issues that arise from such implicated practices of artistic mediation [1]. This is a familiar condition — a human (and posthuman) condition — the same, in a sense, as that which produces the anxiety I feel as I type this essay — using a propriety word processor on a programmable machine costing the better part of an average month’s wages, while all the time it is wired into a network where information and exchange are, themselves, commodified. I interject these prefatory remarks in order that you will share my anxieties, because they are important in themselves and also because I take them to be productive. We make a choice to use these media and it is useful to remain aware of the implications of our choices. I was invited and I agreed to produce work inside an elaborate technological system whose accessibility to public space is problematic, to say the least. Moreover, this question of accessibility is just one aspect of VR technologies’ socio-political context. Once again, these are not unfamiliar conditions of artistic production. Artists still write operas and make films that call for relationships with major industries and massive cultural formations, both of which yield an all but unstoppable inertia. Nonetheless, when the media and contexts surrounding us are deemed to be ‘new,’ for whatever reason, we should not miss the opportunity to reassess the implications of our, perhaps, not-so-novel engagements. Please bear this all in mind as we turn to matters of rhetoric and aesthetics via an entirely pragmatic and relatively unschooled phenomenology.

The Phenomenology of Text in Space
Virtual reality Caves are used, typically, for scientific visualization — for the graphical representation of complex objects, data sets, mathematical constructs, etc. in an illusionistic but perceptually immersive three-dimensional environment [2]. The underlying assumption is that insights concerning the structure and characteristics of these objects will emerge from the experience of perceiving and interacting with them in 3-D [3]. There is now considerable evidence that scientific and analytical insights can, indeed, be derived in this way. Due to the nature of the objects concerned — in so far as they are immaterial mathematical constructs and/or only ever accessible to human senses indirectly — any question of a pre-existing phenomenology of these objects is typically bracketed as irrelevant to the scientific purposes they serve [4].

By the ‘phenomenology of an object’ here I mean simply the significance and affect that human subjects ascribe to the object — its cultural form and meaning as attested by a person or by persons who simply (suspending all questions of how or why this happens) encounter and experience the object. Without wanting to enter too deeply into a relatively unschooled theory of phenomenology, I want to contrast its underlying role in the scientific use of the Cave as opposed to its potential use for artistic practice, especially literary practice. In order to generate aesthetic significance and affect, an artist will, typically, be required to engage with the pre-existing phenomenology of whatever objects her work evokes and inflects. For example, as a poet, I write to engage you with a shared and contested phenomenology of language and of written language in particular. In the Cave, for the sake of these engagements — unlike a scientific visualizer — I cannot bracket the phenomenology of the objects I project. I have to ask myself, ‘What is the pre-existing phenomenology of (written) text in space, and what might be its potential in an artificial three-dimensional immersive environment?’

There is no (figurative) space in this essay for an extended discussion of this phenomenology as such, but I will put forward a few thoughts on the subject in order to contextualize what follows. The question I’ve just asked begs a prior inquiry, ‘Is there a phenomenology of text in space?’ I would answer that there is, but that it is, as it were, constrained. Specifically, in the case of text as writing, it is constrained to surfaces. Text, in this form — as perceptual material object, as a composition of printed or inscribed letters — is two-dimensional. It has no appreciable thickness, and rests, third-dimensionless, on a surface whose thickness is, itself, largely non-signifying. It is often desirable that the actual surface of writing be as thin as possible. Of course there are exceptional cases, where we become aware, for example, of a force producing real depth in a surface that bears a carved or incised piece of lettering, or when the mark-making ‘ink’ is ‘thick,’ literally, as well as figuratively.

There are also a wide variety of exceptional cases arising from practices of Concrete poetry and poetics, to which we will return. However, typically and significantly (and that is what counts in phenomenology), text is a matter for surfaces. In so far as it dwells in space, it dwells on surfaces. Usually this is an opaque and resistant surface, through which other physical objects, including ourselves, cannot pass; not, at least, without damaging or destroying the writing. Of course, surfaces that bear writing do themselves have locations in real space. They are, typically, portable and often gathered into piles or collections of paper leaves. In the West, traditionally, the codex book is where writing most often dwells. The codex is bound and bounded, a compact spatial collection of two-dimensional surfaces, housing the writing that our culture deems to be most authoritative. Books themselves are usually also portable. They acquire real, significant thickness through the gathering of what are, after all, only notionally third-dimensionless surfaces and, when books are themselves brought together, they are set out in libraries, where we find the constrained spatiality of accumulated writing exaggerated in an architectural fantasy of endlessly repeating rectangular shapes, receding in perspective to infinity — shelves and bays and stacks and rooms and corridors and floors, endlessly multiplied in the Libraries of Babel. Apart from there, in the World of Letters, writing in real space occurs on signs, on monumental inscriptions, and on objects that require labels, chiefly containers or vehicles. These are all special surfaces which are very much site and context specific, set up to function in an exact and particular manner — to inform, warn, attract custom and attention; to celebrate, memorialize, identify, and so on. Note that only rarely is the text displayed on surfaces in space anything other than functional; such text requires to be inscribed and read out of necessity rather than rhetorical desire.

Constraints on the spatial manifestation of written text has a further implication relating to the characteristics of writing as graphic form. Because writing is typically located either on portable surfaces or on surfaces that are intended to be read in very specific contexts and for very specific purposes, we have a very clear idea of its graphic constituents’ — that is, its letters’ — size. Letters must always be a good size for reading, big enough to allow us to distinguish the differences which constitute symbolic structure, small enough to copy-fit a significant gobbet of text in the available surface area [5]. Letter forms are, in the Structuralist’s sense, arbitrary. To support the differences they establish, they must be relatively complex shapes, and yet they must become familiar, in the culture where they are current, especially to literate viewers, for whom their very differentiated complexities enable reading [6]. Graphically, and in terms of phenomenology, this gives us — culturally, experientially — a vocabulary of graphic forms about whose size we have very specific ideas and expectations. If you see the shape ‘N’ you expect it to be a ‘legible’ size. When you see a ‘♦’ you do not have the same expectations. This phenomenon has, in my view, major implications for an emergent phenomenology of text in space, as I hope to demonstrate below [7].

Screens and Concrete
Our phenomenology of text is not, of course, static. Cultural and technological change leads to material change in culturally recognized phenomena and to changes in our experience of these phenomena. Since the late 1970s a major part of my writing practice has been carried out in programmable media. This means that I produced and produce texts for screens. Screens are a particular case of writing surface and, manifestly, they are shifting our experience of written language, although more slowly than might have been expected. This point is, arguably, underlined by the present-day florescence of the pdf format, which is largely a ‘paper emulator’ [8]. In the early days of personal and networked computing, screens carried text, but did so in a manner that ran counter to the highly developed aesthetics of print and literary culture. Moreover, this occurred at the same time that screens were establishing themselves, in the form of television, as the visual business end of receivers for broadcast representations of distant real-world spaces. One surface of ‘the Box’ still typically displays visual representations of other places. To do so it employs conventional, illusionistic spatial representations, generated by and adapted from lens optics, photography and film. Today, when we use a computer screen, not only do we expect it to be capable of these visual representations, so familiar from film and television, we also demand a fair degree of spatial representation in our software tools, even when all we are doing is, for example, typing. As I write this, it is in a word processor whose paraphernalia are drawn and colored with all kinds of visual techniques to indicate a shallow but perceptible ‘depth.’ Moreover, I type in a ‘window’ that is ‘in front’ of other ‘windows’ and ‘on’ a ‘desktop’ where ‘icons’ lie suspended and awaiting my attention. All of which (perhaps unnecessarily in this specific context) complicates the phenomenology of writing as a matter of surface, without fundamentally altering or inflecting it.

The purpose of this apparent digression is simply to suggest that even the flat (third-dimensionless) screen of the programmaton, because of its increasingly implicated relations with film and television, invites our engagement with the illusionistic, naturalistic representation of space, and this continues to have implications for any aestheticized presentation of text on the screen, on the making, that is, of screen-based literal art. To be more specific, it has implied (amongst other things) another and, for some readers, a promisingly ‘new’ relationship with the phenomenology of text in space, all without directly addressing the issues and problems as such. Which is, of course, precisely what I am attempting here. The Cave, as the monitor of a graphics-oriented computing system, can be seen as a special type of screen. Indeed, any immersive VR device points to the ambiguity of the display monitor’s role in popular or performative computing. Is the display truly being used as a monitor of the programmaton’s symbolic processing, or is it a window on computing’s attempts to match and then exceed (through the incorporation of transactive or so-called interactive facilities) the illusionistic simulations of film and television? Are not the latter the real virtual of Virtual Reality?

I would argue that the effect on literal art and poetics in new media — the effect of this failure to address directly the phenomenology of text and the ambiguities of the display media — has been to push literal art in the direction of a Concrete poetics. By Concrete, I mean a poetics in which written language — letters and words — take on characteristics of other kinds of objects in order to generate rhetorical and aesthetic affect and significance. Any anthology of Concrete poetry will provide typical examples. Here, we will briefly discuss two works in immersive VR and a tour de force of screen-based textual animation.
Jeffrey Shaw’s The Legible City is something of a classic in the field [9]. The work is not shown in a Cave. Rather, it is projected large in front of its viewers using illusionistic 3-D graphics in order to give the sensation of entering the spaces it delineates — a collection of selectable cities, where words take the place of architecture. The viewer/user navigates their chosen legible city by riding a real-world bicycle interface which guides their point of view through the streets and avenues defined by the letter-buildings. The work thus involves a Concrete poetic in which letters and words taken on the characteristics of buildings in the illusionistic space. There is a simple relationship between the words and the buildings/locations. It is not entirely clear whether you are supposed to navigate for the sake of reading or read for the sake of navigating, or whether (as I did when I once saw the work installed) simply treat the letters as building simulations and then test things like the ability to pass through the ‘walls’ of the letters. My point is that the work is only minimally engaged with a pre-existing phenomenology of text in space. It works by making Concrete poetry’s rhetorical leap of faith — if language behaves like an (architectural) object, how will this make us feel? What new poetry will it give us? In so far as The Legible City does address the phenomenology of text, it does so in problematizing its surfaces. What The City addresses phenomenologically is a potential for us to virtual-physically interact with the surfaces of letters themselves (something which we cannot do in our existing phenomenology of text). It does not deal as interestingly with surfaces of inscription, which are simply dispersed, in The City, to its linear, roadside chains of literal facades. Reading these is basically the same as reading a series of posters or signs on a virtual urban road trip.

Screen, by Noah Wardrip-Fruin and others, is one of the few works for immersive VR which is all but entirely driven by textuality [10]. The work is interesting for many reasons, but in terms of our present discussion, I want to note that the drama of Screen relies, at least in part, on a Concrete poetics. In my reading, as a text Screen belongs both to a tradition of literary prose fiction and to that of the prose poem. Running counter to the expectations generated by VR, the text of Screen is initially projected flat and depthless (like a large-scale inscription within a mausoleum in the real-world phenomenology of text) on the three vertical walls — left, right and front — of the Cave. The text is voiced for the viewer, as if it were being read as an extract from some longer narrative. The tone of the writing engages a poetic sensibility and use of language, directing our critical attention through literary fiction into the realm of the prose poem. Then — since Screen has a temporal, dramatic and even lucid trajectory — the words of the text exhibit behavior. They detach themselves from their surfaces of inscription and drift out towards the reader. The reader is able to knock them back into place, playing them like slow shuttlecocks. Eventually the pace at which the words detach themselves becomes too great to deal with and they overwhelm the reader and her reading. Once again, in the dramatic structure of Screen words behave like playable objects, while the phenomenology of writing and reading remains familiar.

Brian Kim Stefans’ The Dreamlife of Letters makes a valiant, self-consciously Quixotic attempt to exhaust the potential of Concrete poetics on screen [11]. Stefans is one of the few true poet programmers who makes some portion of his work in new media. In practice, he is a poet first. Given Stefans’ critical engagement in the field, I surmise that Dreamlife was, in part, a response to much of the dehistoricized Flash-Concrete that proliferates on the Web [12]. In contrast, Stefans works with an awareness of the wide-ranging tradition of Concrete poetry from Noigrandes to b p nichol and beyond. Moreover, underlying the surface display of dynamic object-like literal behavior, there is a highly serious engagement of his dream-literal language with a poetics that is properly a function of language as symbolic — since Dreamlife is in part a response to writing by Rachael Blau DuPlessis and Dodie Bellamy. Finally, the Concrete excess of Dreamlife is also a tropic rhyme with Stefans’ investment in the Carnivalesque. Its exuberance is not a reductio ad absurdum, but a gesture to poetry inviting it to a masked festival of surface display. Having given Dreamlife its substantial due, it remains to point out that its literal surfaces do not address the phenomenology of text in space, which concerns us in this essay. Apart from in a few exceptional effects, Stefans’ dreaming letters gather, disperse and move in, over and through a Flatland that is, in fact, very familiar as a reading surface, opaque and stationery in front of us, a surface into which we do not enter other than through our interior processing of the letters’ symbolic content; not, that is, through their presentation to our perception and immediate experience.

The point of citing these contextualizing examples is to contrast their deployment of screen-based presentation media with one that directly engages a potential phenomenology of language-as-such in (illusionistic) space. While many examples can be found of Concrete poetics worked into the textual practice of new media, it is much harder to point to direct engagement with a phenomenology of language in space.

I want to cite one other example of practice that I identify as an important engagement related to language and screen. This is the unacknowledged prehistory of textual animation as pioneered in the art of film titles, which I also believe to be the first medium in which words moved [13]. While the vast majority of film titles are instances, at best, of subtle and conservative design, there is a tradition of innovation, and one of its most important exponents — the first acknowledged artist of film titling — is Saul Bass [14]. I bring Bass’s work into the discussion at this point precisely in contradistinction to the role of Concrete poetics. Despite the fact that Bass’s work emerges from design as opposed to fine art or literary practice, I would argue that the film titling that made his name is a groundbreaking engagement with the materiality of language as such in what was then still a new medium for text.

In his most innovative work Bass used the paratextual features of letter and word forms both to define graphic space and to dwell and move in and over the surfaces of the illusionistic naturalism within the already well-developed visual rhetoric of narrative cinema. Bass used letters to shape space, or he located them on spatial surfaces in ways that remained faithful to the phenomenology both of objects in the visual space of film and of language itself as I have sketched it above. Two examples are most pertinent to the following discussions. In Bass’ masterpiece of titling, for Hitchcock’s North by Northwest, the titles are dynamically displayed on a rectangular surface perspectivally presented at an oblique angle to the camera [15]. This resolves to the surface of a glass-fronted building, where the credits are still successively ranged, but now inscribed on a surface that is photo-naturalized and, illusionistically, a part of the filmic real world. The letters and words do also exhibit some Concrete-style behavior (they rise and fall like elevators seen through the building’s glass), but this is less important and less striking than the interaction they establish between language and natural-world surface-in-space.

Secondly, in his titling and credits for West Side Story, Bass used the filmed surfaces of signage and graffiti for his surfaces of inscription in a more or less perfect rendering of the pre-existing phenomenology of language in space, but enhanced and aestheticized. In this, he highlighted a major site for further work and experimentation, and in certain of Bass’ followers there is evidence of continued practice; for example, Kyle Cooper’s titling for Se7en where the titles are scratched into the actual film stock.

As yet, both this aspect of Bass’ practice and its underlying rhetoric have been little-acknowledged. In part, of course, this is because of the inaccessibility of appropriate media to appropriate artists. The increased accessibility of immersive VR has potential to change this state of affairs. In Bass’ later work he reverted to the dominant mode of screen titling in which letters and words ‘float over’ the visual world of the film on planes that are, conceptually, in an entirely different space to that of the underlying photo-naturalism. This mode is also relatively familiar in new media work with language in the form of writing that is, basically, illustrated by visual and audio material rendered in new media. In fact, the majority of existing poetic engagements of text and new media can, I think, be characterized as either new media Concrete or filmic illustration. Examples of work in which aestheticized experiences of space and language work together as such are relatively difficult to find [16].

On the Threshold of the Cave
When invited to begin making work in the Brown University Cave, I was already wary of the problems and questions that are raised and discussed in this essay, but had not developed related arguments or provisional answers. A narrative of our writing in space, still very much in progress, seems to me a good example of the way in which experimental practice can, at times, generate constructive contributions to our engagements with critical culture and theoretical issues, some of them entirely unanticipated.

My work in writing for programmable media has, in a number of instances, involved designing and implementing a conceptual topology for textual structures. Specifically, I have recognized that the programmability of both compositional and delivery media allows for the disposition of texts in an ordered manner such that, for example, media can represent structural interrelationships between the texts, and that such an arrangement may be most easily figured as spatial.

In noth’rs, for example, the textual fragments populating the piece as a whole are conceived as arranged in circles or rounds, four at a time [17]. When the system is in any one of a very large number of potential states, four textual fragments are immediately and consistently accessible. One — at a particular ‘position’ on the circle — will be shown on the screen. Using the computer’s navigation keys, the view of the text changes, moving from the reader’s textual starting point to the adjacent text, doing so ‘through’ a series of transitional texts until a new quadrant of the circle is reached [18]. Other (command) keys may be used to change the state of the piece, but at any one time, if the appropriate navigation keys are used, the reader’s view moves ‘around’ the circle from one to another of four texts. noth’rs is entirely text-based and accommodated to display on a flat screen; when a ‘movement’ is made, one table of letters is simply replaced by another. There is no visual representation of the conceptual spatial arrangement of the text. This must be imagined by the reader rather than (illusionistically) perceived. noth’rs does have an audio channel in which dynamics are associated with one aspect of proximity. However, in this work the sound ‘source’ is loudest in between the nodal (natural language) texts and therefore, in a sense, runs counter to an experience of a naturalistic space underlying the piece.

In riverIsland, a conceptual literary topology is more fully elaborated and is, moreover, associated with (distorted but nonetheless recognizable) visual imagery derived from naturalistic landscape [19]. In one dimension of this piece, 16 short lyrics are arranged in a circular formation. By navigating left and right, the reader moves from poem to poem. When the reader is ‘in front’ of a poem, its text is displayed in natural written English and a recital of the lyric loops in the audio channel. As the reader ‘moves away’ from the poem the dynamics of the recital fade and a series of algorithmically generated transitional texts — transliteral morphs — are displayed on the screen. Eventually, as the reader reaches the ‘next’ poem, the sound of its recital reaches full volume and its natural English text is displayed. The audio dynamics are stereo, reinforcing these spatial conceits. There are many other aspects of the piece with which we will not complicate the present discussion.

Despite its more elaborate spatiality, the texts of riverIsland are, as with noth’rs, presented on a visually flat screen with no representation of depth or implied relative position through, for example, a perspectival distortion of the graphics of the text. Instead, media other than literal, textual media are used to suggest and reinforce an illusionistic spatial experience, one that remains conceptual in relation to the visually perceptible materiality of the language itself. When I came to make work in the Cave, there was, therefore, an obvious first step to make: use the Cave’s immersive 3-D graphics to delineate a topology, a shaped space in which text is systematically disposed. I decided to make a shape similar to that implicit in riverIsland but using source material from noth’rs. The shape resolved itself into a doughnut-like torus and the working title of the piece takes its name from this figure. In the Cave piece, a torus shape is defined implicitly by 16 vanes of text on planes arranged around the doughnut’s central, circular cavity. The vanes are aligned with a series of radii extending beyond the bounds of the interior circle, such that the inscribed planes are like upright cross sections through the ‘dough’ of the ‘nut.’(see figure 1, above) Although tricky to describe, illustrated, I believe this is an arrangement that is fairly intuitive and easy to understand. It provides an appropriate abstract spatial model for a topology very similar to the one that was implicit in riverIsland. Now we come to our narrative of experiment and discovery.

Applied (Literal) Phenomenology
The decision to build a torus seems to have been contingent and intuitive, but at the time it was made I was already concerned with the questions of a phenomenology of text in space. Simply to write texts on the implied surfaces of the torus-defining vanes seemed an inadequate engagement with those questions. The inscribed vanes would be like billboards or thin monumental inscriptions set out in an abstracted minimal architecture. This was not enough. The letters themselves, I felt, should address both their existing phenomenology and its potential. As we’ve discussed, letters have no thickness. They also have only a single right-legible surface — the one that is presented to us as we read. Few letters are left-right symmetrical. In a well-designed typeface it is arguable that none are symmetrical in this sense (with the possible expectation of ‘o’ in alphabetic scripts). I decided that the letters defining the torus world would always face the Cave’s primary point of view [20]. The letters of a plane- or surface-defining text are ‘pinned’ to a particular point on this implicit surface but pivot freely in any direction in response to the position and orientation of the primary point of view. The tracked viewer can never get ‘behind’ a letter. She can never, in fact, be other that directly facing a right-reading letter, which is also always ‘vertically’ upright in relation to her point of view. In other words, letters stay put but always turn or rotate to face their reader in the 3-D graphics world of Torus. This immediately shifts our phenomenological expectations in relation to text in space. Only if we are facing the ‘front’ of a plane on which there is a text, will it be readable like a sign or billboard, and give the familiar impression of such a billboard. If we move to the side and finally to the edge of the plane, the letters turn to face us, with nearer letters appearing larger, and partly occluding those behind. If we move to the ‘back’ of the plane, the letters will still all be facing us, but the text will be a ‘mirror’ text, although one composed of right-reading letters rather than mirror letters [21]. If, at this point — ‘behind’ the plane of inscription — we were to turn upside down, we would see the text once more as right-reading.

We have not yet taken up directly the question of the reader’s movement and control of position in the Cave. Using a wand, a tracked reader in the Cave can move or, more dramatically, move the ‘world’ (as it is known) in relation to their point of view, more or less at will, depending on the world’s programmed constraints. In the case of Torus, I wanted the reader first to see the torus shape as a distant structure below the level of their position in the conventional blackness of space. The reader sees the torus as a doughnut shape suggested by its vanes, which, themselves, are delineated by letterforms ‘pinned’ to the their surfaces. All of these letters, rendered in slightly differing sizes according to rules of perspective, will be turned to face the reader’s point of view, as outlined above. The reader soon discovers that she can move (‘fly’) down towards the torus (or, if she prefers to see it another way, bring the torus towards her position of stasis). She is also able to move into the implicit body of the torus, then turn to face, read and move through its vanes. The reader’s circular movement inside the torus from vane to vane is conceived as the closest approximation to conventional reading that the system provides.

In making this global movement, the reader also passes through the images of the letters and words suspended on the vanes, their faces turning as she does so, sunflower-like, towards her passing point of view. Of course, these movements and the implied relationships to surfaces of inscription (both the letters’ surfaces and the surfaces of legible text) are far from being real-world phenomena. There is little in the system that might be characterized as simulation — only the relatively easy construability of an abstract architecture combined with a spatial imaginary that has been elaborated by the history of flight and the beginnings of human experience of ‘space’ in its other sense.

Apart from these framing metaphors, there is no simulation of a pre-existing phenomenology. Instead we are confronted with experiments towards an emerging or potential phenomenology based on an extension of perceptual experience, specifically the perceptual experience of writing. Literal material is disposed and given systematic behavior, and this is enough to define a space-time for the reader-viewer. Given the strangeness of these circumstances, they are strangely familiar. We might pause to ask how is this so easily possible? Why shouldn’t the fact of the graphics’ linguistic materiality counteract or nullify an illusion of spatiality? Quite the contrary, I argue below that certain characteristics of written matter reinforce and enhance the illusion. For the moment, however, I would simply like to recall that the literal disposition and behavior of the Torus is not a function of any form of Concrete poetics. The behavior of the literal material refers to and plays off a pre-existing phenomenology, but one that is derived from selected aspects of the pre-existing behavior and characteristics of written language itself. It is not an application of the behaviors and characteristics of other objects and media to linguistic phenomena.

Bugs as Futures
At this point in the underlying narrative, we encounter the swerve of chance, the clinamen so fervently desired by many radical formalists, myself included. There was a known anomaly in the graphics system of the Cave, not really a bug, but more a matter of a default configuration in rendering that produces counter-intuitive visual effects. The chance occurrence of this bug allowed me to clarify specific thoughts relating to the some of the questions that we are exploring. The effect of this anomaly was that, in certain contexts, the surfaces of conceptually and perspectivally distant objects in the Cave are rendered over the surfaces of closer objects in terms of transparency/opacity [22].If letters were all rendered in the same surface colour with no lighting effects or without anti-aliasing or similar sophisticated edge rendering techniques, then this ‘bug’ would not have been noticeable. However, even a smaller, conceptually more ‘distant’ white letter rendered ‘over’ a larger, ‘closer’ white letter will be visible because its edges are made visible by the graphics engine’s subtleties.

As explained above, I wanted my letters in space always to face the primary tracked point of view. If the tracked reader is positioned at the edge of a plane of letters and she turns to face the plane edge-on, the letters turn to face her. Their images overlap, occlude one another — partially or wholly — and recede in view since the majority of them will be successively more or less distant. ‘Normally’ the surfaces of the larger closer letters would cover the more distant smaller letters. However, because of the ‘bug,’ smaller letter outlines are clearly discernable within but ‘over’ the forms of the nearer letters.

Given these circumstances, and because, I believe, all the letter forms are familiar — both visually and symbolically legible — and because we know what their relative scale ‘should be,’ this produces a striking and somewhat bizarre visual illusion.



Figure 2 © John Cayley

Figure 2
Copyright © John Cayley

We assume that even though the smaller letters are rendered ‘over’ the larger ones, they must be more distant (as in fact they are in the conceptual topology). Thus, what we see is a very deep and narrow corridor formed from letter shapes, with the most distant smallest letters visible in completely edged outline, apparently farthest off, as if inscribed on a tall, thin distant end ‘wall’ of the corridor (Figure 2, see above). Moreover, the reader is able to move ‘into’ the corridor formed by this plane of letter shapes.

This powerful perceptual experience is demonstrable and repeatable, despite its artificiality and strangeness. The question arises, why should this phenomenon be so immediate and effective? As we asked above, why doesn’t the linguistic materiality of the graphic forms and structures run counter to their visuality, counter to whatever illusion of space may or may not be generated? At this point I began to formulate a hypothesis: literal forms are highly effective for delineating space in immersive virtual environments.

As suggested above, I believe that there are a number of quite common sense related reasons for this. Letter forms are relatively complex and the differences instantiated in this complexity are necessarily and systematically significant. Letter forms are both complex and arbitrary but they are also familiar and rational seeming. They are instantly recognizable and encountered with remarkable, perhaps unique, frequency by any subject in a culture where the set of script forms in question is current, particularly, of course, by subjects who use these forms to read and write [23].

Moreover we have a very well entrenched set of expectations in relation to the relative size of these arbitrary graphic shapes. If we believe them to be on the same plane, we expect literal elements all to be of a similar or commensurate size (equal constituents of the same text) or, if they are of different sizes, we expect there to be a paratextual reason for this (because, for example, the letters are part of an emphasized word or a title). If we see words or letters which are larger than the letters surrounding them and there is no paratextual reason for this, we are likely to think that these letters are closer to us than any surrounding smaller letters.



Figure 3 © John Cayley

Figure 3
Copyright © John Cayley

A whole range of experiments in perception and cognition immediately presents itself to me and I hope that some research in this area might be inspired by this account. I am not an experimental psychologist however and my remarks remain at the level of informed speculation. Nonetheless, here are two simple visual exhibits which emerged from our time in the Cave and from which readers may begin to draw their own tentative conclusions. To test our theory that letters are relatively good at delineating space, at one point we simply replaced the letter-shapes used to imply the torus with repeated instances of a single simple abstract shape — with spheres or rather the somewhat irregular approximations of spheres native to the graphics engine in a relatively lo-resolution mode. Here are two rendering of the torus, one in letters (Figure 3, see above), one in these spheres (Figure 4, see below). Which gives the stronger visual impression/illusion of a three dimensional structure?



Figure 4 © John Cayley

Figure 4
Copyright © John Cayley

Or consider what happens to your sense of the relative distance of the various visual elements in the following simple diagrams [24]:

<visit: http://www.shadoof.net/in/?nf/index.html
... to explore the diagrams>

If these effects are conceded, it follows that literal materiality can have powerful influence on the shaping of immersive 3-D space and the structures within it. What I take to be happening is that the pre-existing, enculturated phenomenology of textuality, when disposed in this illusory space, struggles to maintain its real-world phenomenology, with and against the graphic world’s programmed structures. A relationship of mutual interference and influence develops in which the phenomenology of text shapes the reader’s experience of space while, at the same time, of course, spatiality works to alter our phenomenological experience of text in its new, artificial environment. For the moment, in the present context, we will concentrate on and conclude with the ways in which text may shape space.

Surface Spaces
The Cave’s rendering anomaly had other, related effects which tended to reinforce the speculations sketched out here and lead us to the final part of this case study describing the production of a separate small work, Lens, both in the Cave — where it is more fully realized — and as a interactive QuickTime maquette, appended to this essay.

... to read and experience the maquette>

In one development version of Torus, letters of different colours are used and it was noted that, as expected, ‘distant’ letters will render over closer ones in the anomalous configuration. If the distant letters in question are dark in colour and the nearer letters light, then, effectively, the surfaces of the nearer letters are transformed, by the anomalous rendering, into surfaces of inscription for the distant letters. If the overall background colour is dark (black by default, as in the existing Cave version and also the initial state of the present QuickTime maquette) this has a further effect relating to legibility and strategies of reading. Dark and distant letters on a dark background are difficult to read. On a lighter background they may suddenly become legible. If the lighter background happens also to be the surface of a letter that otherwise seems to be perceptually close to the reader (it is closer in the conceptual topology of the graphic world), a strange counter-intuitive effect is produced when the dark letters stray into the region of light — a literal surface becomes a surface for inscription/reading and the spatial relations between the textual surfaces inverted. The surface of the nearer letter may also, as we shall see, become a full-blown 3-D space within which the more distant letters appear to be disposed.

In the appended maquette, which uses no actual 3-D rendering and in which illusory visual distance is represented only by the sizes of its various texts, these effects can nonetheless be demonstrated. Distant texts, two dark- and two light-coloured, drift in the screen’s blackness. There is also, at first, a ‘lens’ word rendered in larger white letters. The reader can move this ‘lens’ by dragging and scaling it using command keys. If the lens itself is zoomed-in so as to become (illegibly) large, the surfaces of one or other of its constituent letters can then be used as a reading surface for the more distant darker texts and this makes them suddenly legible, as well as subverting our assumptions about their relative distance.

In the Cave version of Lens, the effects are far more striking, disturbing and spectacular. The letters of Lens obey the constraints set up for Torus — their surfaces turn towards the tracked point of view — and the textual objects in the piece are fully 3-D as is the space itself. The lens text can be moved in relation to the reader’s point of view, drawn close or sent out amongst the distant darker texts, like an investigative spotlight. Most spectacularly, because of the immersive characteristics of the Cave system, the literal surface of the lens object’s letters can be, as it were, moved so close as to touch or even pass ‘behind’ the reader’s body and point of view. The surface of a lens letter can even be brought into the eyes of the reader. When this happens, the eyes seem to be flooded with the white light of this literal surface and the most spectacular spatial inversion/subversion occurs. The whiteness becomes a 3-D space. In fact it becomes the enclosing 3-D space of the Cave, taking the place of the dark space previously inhabited by both reader and the various textual objects only a moment before. The distant dark blue texts still drift in this space, but now they do so, distinct and legible, in a space of light and clarity. If the reader then moves the surface-literal lens-light ‘out’ of her eyes, the enclosing space, as suddenly, reverts back to darkness [25].

This piece, a relatively simple system, demonstrates and — perceptually, phenomenologically — concentrates the various effects we have been introducing and discussing. The space itself in Lens is defined by letter forms alone — simple planes of text that appear to be drifting a space of indeterminate size. It is remarkable that this illusion can be generated so convincingly with so small a number and vocabulary of graphic objects. Once more, we might surmise that this is precisely because of the familiarity and potential linguistic function (subverted by ‘distance’ and ‘darkness’) of the constituent objects. The size plus the illegibility-in-context of objects we nonetheless recognize as potentially legible equals/reinforces the illusions of distance and spatiality. Literal materiality conspires to define space and does a good job of it. The counter-intuitive rendering of transparency then subverts what is a powerful pre-existing phenomenology, both that of spatial perception and also the related phenomenology of text. The surface of a letter — something that we are used to experience both as and on an opaque persistent surface — becomes not only a surface of inscription, a surface for other letters, a surface that allows reading, in itself, it opens out to space within which texts can dwell in legible clarity. In fact, moreover, the instantaneous transformation of potential into actual legibility is, arguably, the very event in Lens that produces the spatial inversion, where the perception of enclosing dark space is instantly replaced by enclosing light space. Literal graphic materiality is able to entirely and suddenly transform spatial perception and, at the same time, it creates an entirely new space for itself, for inscription and for reading. It creates the potential for a new experience of language.



Figure 5 © John Cayley

Figure 5
Copyright © John Cayley



Figure 6 © John Cayley

Figure 6
Copyright © John Cayley

Brief Provisional Concluding Remarks
Myself, I am convinced by my experiences in the Cave, that literal materiality is good at defining space — remarkably, significantly good. It provides us with forms that are abstract, arbitrary, complex, and familiar precisely because it embodies the symbolic. As such, literal materiality gives us — quickly and efficiently — a great deal of visual information which can be applied by 3-D graphics engines to the generation of an illusory and — given appropriate delivery media — immersive spatiality.

Specifically, I believe, literal materiality provides an important vehicle for the investigation of the phenomenology of surfaces, surfaces in general, but also, of course, surfaces of inscription and reading — i.e. symbolically implicated surfaces [26]. We have not finally engaged with questions of any ultimate aesthetic, cultural or social values that may arise from investigating these phenomena further. In their instances of practice, Torus and Lens proffer their aesthetic values. They, at the least, pretend to bear a certain measure of significance and affect. Any judgment concerning the potential of this work’s attestations is for others to make. It remains difficult, if not impossible, to answer the question of whether or how far such a potential provides justification for the use of media, which are relatively inaccessible and technologically implicated.

What I can predict is that three-dimensional immersive representations will become more and more current as technology develops, and that language can and will have a role in this virtual reality, however, ultimately, it is mediated. On the evidence of the experiences narrated here, symbolic, literal materiality will not only exist in our virtual spaces, it has the ability to contribute to their very constitution, to shape and define these spaces. This is something that should be recognized and used in development now, not bracketed or deferred in pursuit of the more banal representations of space that arise out of the simulations of a purely visual imaginary.

References and Notes
Cayley, John. “Bass Resonance” in Mute January 2005, pp. 22-24.

---. Noth'rs 1999-, http://www.shadoof.net/in/nothrs.html, last accessed February 2005.

---. Riverisland 1999-, http://www.shadoof.net/in/riverisland.html, last accessed February 2005.

Cruz-Neira, Carolina, Daniel J. Sandin, and Thomas A. DeFanti, “Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the Cave” Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Siggraph ’93) (New York: ACM Press, 1993) pp. 135–42.

Manovich, Lev. The Language of New Media (Cambridge: MIT Press, 2001).

Shaw, Jeffrey. The Legible City 1989, http://www.jeffrey-shaw.net/html_main/show_work.php3?record_id=83, last accessed February 2005.

Stefans, Brian Kim. The Dreamlife of Letters 2001. Flash-based website. UbuWeb, http://www.ubu.com/contemp/stefans/dream/, last accessed September 2003.

---. “Stops and Rebels: A Critique of Hypertext” in Fashionable Noise: On Digital Poetics (Berkeley: Atelos, 2003) pp. 61-169.

Wardrip-Fruin, Noah “Playable Media and Textual Instruments” Netzliteratur: Umbrüche in der literarischen Kommunikation (Universität Siegen, 2004).

---. Screen 2003, http://hyperfiction.org/screen/, last accessed February 2005.

1. I would like to thank Brown University’s Program in Literary Arts for the opportunity to work and direct research in the University’s Cave during the spring of 2004. In particular, I would like to thank Professor Robert Coover, who invited me to take part in the program in this way. While at Brown I benefited from discussions and other interactions with, amongst others, Noah Wardrip-Fruin, Roberto Simanowski, Talan Memmott and Bill Seaman (at the neighboring Rhode Island School of Design). Dmitri Lemmerman was my main collaborator on the projects outlined here; Thomas Owen and Jason Moore helped with sound design for Torus.

2. A good technical description of the basic working of Brown’s four-wall cave can be found in Carolina Cruz-Neira, Daniel J. Sandin and Thomas A. DeFanti, Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the Cave”, in Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Siggraph ’93) (New York: ACM Press, 1993). Here is a lay description from Noah Wardrip-Fruin, “Playable Media and Textual Instruments”, in Netzliteratur: Umbrüche in der literarischen Kommunikation (Universität Siegen, 2004). “Brown’s VR chamber is similar to the University of Illinois’s CAVE — a virtual environment that shows three-dimension images while allowing users to continue to see their own bodies, and that does not require users to wear encumbering equipment (unlike head-mounted displays, which are essentially blindfolds with televisions inside). Brown's Cave is an eight foot cube, missing its top and one side, and its walls and floor are screens. Projectors are pointed at each screen, and they alternately project streams of images meant for the user's left and right eyes. The user wears shutter glasses that alternately occlude the left and right eyes, in synchronization with the projectors. The result is stereo VR — 3-D vision of computer-generated imagery — combined with the physical presence of the people and objects in the Cave.”

3. Caves are also, of course, used for projects best seen as relatively straightforward primarily visual simulation of space — architectural and/or natural. I do not discuss such simulations here.

4. At least as far as immediate scientific goals or results are concerned. A scientific model does not necessarily have significance as such, whereas an artistic form (e.g. a sculpture) is, arguably, its proper meaning. As such the form may encompass any representational function it is deemed to possess (which may, in certain cases, constitute only a part of its total ‘meaning’).

5. For the gobbet to be functioning as writing that is. It may however be a ‘representation’ of writing rather than writing per se. This was pointed out to me by Robert Harrist, the historian and critic of Chinese art and especially calligraphy — a representation of writing must be illegible, otherwise it is writing (E-mail correspondence).

6. Of course, letters are also familiar to the illiterate viewer who comes from a culture where the letters in question are current. One can postulate an experimentally discernable difference in the degree of familiarity and, of course, in the expectation of legibility, which might affect the phenomena I discuss in this essay.

7. There is little discussion in this essay of the audio experience of language in the Cave. We are chiefly engaged with a phenomenology of writing as graphic representation in 3-D space. For a more complete exposition of our subject the spatial phenomenology of spoken language would demand commensurate consideration. I have placed some initial thoughts in this footnote. If we take, as I believe we should, a Derridean view of speech — particularly of aestheticized or performative speech — in which it is seen as special case of inscription, an aspect of grammatology, we may be helped to see that spoken language is also highly constrained in its spatiality. It emanates from a point, from a speaking subject. It is directed — the mouth faces and projects in a particular direction — but it also propagates in all directions from the single point of emanation, though less strongly in the directions other than where it is directed. Speech has, itself, no phenomenological extension in space. It propagates instantaneously, as experienced by any hearer, from a point of emanation to points of audition and potential symbolic processing. Effectively, therefore, as it comes into being, spoken language exists simultaneously at an indeterminate number of spatial points — the point of emanation plus any number of points of audition. An aestheticized engagement with audible language in an immersive spatial environment would have to take account of these and other considerations in a similar way to that proposed for the graphic instantiation of writing in this essay.

8. The phrase is Ted Nelson’s, who, in his recent public pronouncements (I last heard him lecture at Oxford University, 10 July 2003), takes furious pleasure in pointing out what he sees as a supremely ironic and wasteful paradox of textuality in new media.

9. See: Jeffrey Shaw, The Legible City 1989, http://www.jeffrey-shaw.net/html_main/show_work.php3?record_id=83, last accessed February 2005. There is significant discussion of this work in Lev Manovich, The Language of New Media (Cambridge: MIT Press, 2001).

10. Noah Wardrip-Fruin, Screen 2003, http://hyperfiction.org/screen/, last accessed February 2005.

11. Brian Kim Stefans, The Dreamlife of Letters 2001, flash-based website, UbuWeb, http://www.ubu.com/contemp/stefans/dream/, last accessed September 2003.

12. See: Brian Kim Stefans, Fashionable Noise: On Digital Poetics (Berkeley: Atelos, 2003). Especially: Brian Kim Stefans, “Stops and Rebels: A Critique of Hypertext” Fashionable Noise: On Digital Poetics (Berkeley: Atelos, 2003).

13. I have written slightly more fully on Bass in John Cayley, “Bass Resonance” Mute January 2005. This short article was produced in response to a one-person show at London’s Design Museum, 17 July –10 October 2004. A version of the article is also available online at the Electronic Book Review http://www.electronicbookreview.com/thread/electropoetics/dynamic (last accessed December 2005).

14. Saul Bass was the first film title designer to be given a screen credit by the Director’s Guild of America (for Preminger’s ‘Carmen Jones’ 1954).

15. At http://www.notcoming.com/saulbass/caps_nxnw.php (last accessed December 2005) you can view a ‘stop-frame’ extract based on the titles and a brief appreciation by Beth Gilligan, North by Northwest: the Title Credits, notcomimg.com, 2004.

16. One important example I do not discuss here is the subgenre of (usually java applet based) visual thesauri (http://www.visualthesaurus.com/). Although, as Noah Wardrip-Fruin puts it, ‘What started out being presented as art is now being presented as software’ (e-mail communication 31 January 2005) Wardrip-Fuin also pointed me to a review of the previous version of the visual thesaurus at: http://intelligentagent.com/archive/vol2_no3_vis_thes.html (last accessed February 2005). In 2003-2004, Talan Memmott worked on a piece in Brown University Cave called A Semiotic Oscillator in which viewers could engage with a branching structure of words and phrases spatialized similarly to a visual thesaurus although, ‘In the end it would have been quite different from the Visual Thesaurus, or some such ...’ (e-mail communication 31 January 2005) since it would also have a separately structured harmonic dimension and visual material in an entirely different register.

17. John Cayley, ‘Noth'rs,’ 1999-, http://www.shadoof.net/in/nothrs.html, last accessed February 2005. This work is described on and can be downloaded from my website, http://www.shadoof.net/in. Much abbreviated, earlier versions of noth’rs appeared on the CD ROM which accompanies Performance Research ‘On Line,’ Volume 4, Number 2 (Summer 1999), edited by Ric Allsopp & Scott deLahunta); and on the web at Riding the Meridian http://www.heelstone.com/meridian/cayley.html. An initial performance version was shown at Digital Arts and Culture 1999, Atlanta Georgia, 28-31 October 1999.

18. I make a distinction between nodal and transitional texts, but readers interested in these matters should note that noth’rs and others pieces like it are better seen in terms of applied grammatology, as figurations in process of the Derridean notion of ‘différance.’ The text offered to the reader is all of its identifiable elements, its transitional states and the processes in between.

19. John Cayley, riverIsland 1999-, http://www.shadoof.net/in/?riverisland.html, last accessed February 2005.

20. In the Brown Cave and for the programming of Torus there is only one tracked point of view (which I also call the primary point of view). Other viewers can be present in the Cave and they see what the tracked viewer sees, slightly offset by their position in the real space of the Cave. But they do not have to be looking in the same direction as the primary point of view. In contrast to the case of a flat screen representation of a 3-D world (everybody sees the same thing) in an immersive 3-D environment, other viewers can, for example, look behind the tracked point of view and see what that viewer would be able to see if they had ‘eyes in the back of their head.’

21. We could, of course, have chosen to make other linguistic elements the graphic atoms of this type of behavior and I am sure experiments making words, phrases, sentences, textual fragments behave as a whole in this way would yield interesting and possibly affective results.

22. Thanks to Stephanie Strickland who helped us to spot this anomaly during a visit to the Cave in spring 2004. Please note also that, apart from the insights derived here, such a procedure points to a general (and not infrequently applied) rule of experimental practice — work out what the underlying assumptions of a media system are and then experiment with the effects of running counter to those assumptions.

23. We might pause to wonder whether letter forms are the most frequently experienced abstract artificial shapes encountered by humans?

24. And notice, again, your implicit assumption that the letters composing each of the words in these diagrams are on a single plane at a single distance from you.

25. We have considered and would like to implement a related Cave work with the similar features, and with the infinite regression of the maquette — in which, as the reader flies down and into the letters of a light text, she discovers dark texts within the space its literal surfaces create, and then within those dark texts, other light ones, and so on, endlessly.

26. Using some of the material in this essay I have already gone on to explore related questions in Writing on Complex Sufaces, a paper presented to the 6th Digital Arts and Culture Conference, which was held at the I.T. University, Copenhagen, 30 November-1 December 2005. There is a printed proceedings. Refer to: http://www.dacconference.org. A pdf of the paper may also be found at: http://homepage.mac.com/shadoof/pdfs/surface4DAC.pdf.

 Author Biography
John Cayley is a London-based poet, translator, publisher and bookdealer. Links to his writing in networked and programmable media are at www.shadoof.net/in. His last printed book of poems, adaptations and translations was Ink Bamboo (London: Agenda & Belew, 1996). Cayley was the winner of the Electronic Literature Organization's Award for Poetry 2001 (www.eliterature.org). He is an Honorary Research Associate in the Department of English, Royal Holloway College, University of London, and has taught and directed research at the University of California San Diego and Brown University, amongst other institutions. His most recent work explores ambient poetics in programmable media, with parallel theoretical interventions concerning the role of code in writing and the temporal properties of textuality (bibliographic links are available from the shadoof site).

Download pdf version here

Citation reference for this Leonardo Electronic Almanac Essay

MLA Style
Cayley, John. "Lens: The Practice and Poetics of Writing in Immersive VR (A Case Study with Maquette)." "New Media Poetry and Poetics" Special Issue, Leonardo Electronic Almanac Vol 14, No. 5 - 6 (2006). 25 Sep. 2006

APA Style
Cayley, J. (Sep. 2006) ""Lens: The Practice and Poetics of Writing in Immersive VR (A Case Study with Maquette)," "New Media Poetry and Poetics" Special Issue, Leonardo Electronic Almanac Vol 14, No. 5 - 6 (2006). Retrieved 25 Sep. 2006 from

Advanced Search
Join LEA's New Media Poetics Discussion Forum and Live Chat today!
Wired Magazine on LEA
  SUBSCRIBE to LEA | Site Guide | Contact  
  © Leonardo Electronic Almanac | All Rights Reserved 1993 - 2006 | Disclaimer | Copyright | Last updated : 22 September 2006
Leonardo Online Leonardo Leonardo Music Journal Leonardo Electronic Almanac Leonardo Book Series Leonardo Book Reviews OLATS Leonardo Activities and Projects About Leonardo