MobiLenin and Manhattan Story Mashup - Two multimodal user interfaces for creating and sharing interactive artistic experiences
Click here to download pdf version. (156kb)

by Jürgen Scheible
Doctoral Student
Media Lab
University of Art and Design Helsinki
Hämeentie 135C
FI-00560 Helsinki,
Finland
jscheib [at] uiah [dot] fi
http://www.mobilenin.com
http://www.mobileartblog.org

Keywords
mobile interaction, public displays, interactive music video, Python for S60, street art, mobile device as art tool

Abstract
This paper discusses two projects where multimodal user interfaces were built from personal mobile phones, a large public display, the web and multimedia data for the purpose of creating and sharing an interactive artistic experience between people. The main focus is on the MobiLenin system, which allows a group of people to interact with an interactive music video shown on a large public display using their personal mobile phones. The proposed system includes a lottery mechanism as an incentive for interaction. The results of a user evaluation conducted in the true environment of use show that the system could offer a new form of entertainment in pubs and other public places.

The second project described briefly is the Manhattan Story Mashup, an urban storytelling game combining mobile phones, the web and a large public display into interactive, collaborative street art. The paper concludes by analyzing the key role of the Python programming language in the successful implementation of the projects. The projects demonstrate how Python can be employed for creative prototyping, enabling people to use their mobile devices as an art tool.

	Figure 1: MobiLenin in real-world setting Copyright © Jürgen Scheible

Introduction
The MobiLenin system focuses on employing a large public display and personal mobile phones for social interaction between people via interactive entertainment. The principal idea is that each user can interact with an interactive multi-track music video shown on the public display (see Figure 1, above). Each user can individually vote for one of the tracks by selecting the corresponding choice from the menu in the mobile phone application, and the track receiving most votes is shown. Thus, the MobiLenin system empowers the users with a joint authorship of the interactive art piece. As an incentive for interaction, the system chooses randomly a winner among the users having voted in each voting interval. The winner receives a coupon on the phone entitling her for a free beer or free pizza.

Previous research on the design and use of public displays for interaction has identified a broad variety of challenges, which mostly pertain to enticing people to interact with them [1]. Churchill et al. [2] found that users needed “constant encouragement and demonstration” to interact with the interactive public display. This means there is need for solutions which make the users feel invited, interested and encouraged to interact.

Public displays and mobile phones make an interesting couple in terms of strengths and weaknesses. While shared displays typically offer greater conceptual power and larger presentation space, they often limit interaction to one user at a time. Mobile devices, on the other hand, disperse control and access to participating users, though limited conceptual power and smaller screen sizes often hinder dynamic interaction. Thus, connecting shared displays to mobile devices is an obvious way to leverage the best of both worlds [3]. According to Paek and others [3], interactive shared displays are most suited for certain types of applications, including “collaborative tools allowing multiple people to contribute to a single goal”, and “arena applications involving competitive interaction” – the MobiLenin system relates to them both.

Artistic Motivation
By using my engineering skills in combination with my skills for producing multimedia content and writing original songs — namely composing music, writing lyrics, producing music and doing live performance as well as video production — I try to leap to new levels in novel interactive systems and interactive art pieces that inspire people by carrying them to new heights of interactive experiences.

My motivation for creating MobiLenin was to give the audience the possibility to engage in a new way in my live show — simply by interacting with the music and video on a large screen. The idea is to enhance people's concert experience by letting them interact with me in the virtual domain (display). Through the virtual domain people can interact with attributes and scenarios (e.g. turn the artist into a skeleton), which can never be offered in the physical domain on stage - simply due to the physical limitations.

I wanted to build a system where on one hand the interaction affordance offers elements of collaboration for large but also small crowds. At the same time there should be a private return channel to people to reward them for their participation in the interaction (e.g. winning a CD). The interaction should be fun and easy to use, and it should trigger social engagement with other people. The system should generate its own unique artistic expression which feeds people's emotion and sympathy. It should also give them a fresh and positive experience that fits to the brand of my music and my appearance.

By providing my own music and producing the music video, as well as designing and programming the client server architecture and the mobile application for the MobiLenin system, I tried to use my artistic approaches to create a unique but in itself a coherent interactive system that meets my explained goals.

The MobiLenin System
In the music video employed in the MobiLenin system, the main character is my alter-ego known as “Lenin’s Godson”, which explains where the name MobiLenin comes from: Mobile and Lenin. The “Ggogogo” song used in the music video is extracted from my published albums, where it is available in three different versions of equal length:

1. A full version with guitar sound and singing
2. A reduced version with guitar sound, but no singing
3. a slim version with no guitar sound and no singing.

The music video comprises of six tracks which are of precisely equal length and they are played in parallel and in sync, whereas only one track is visible at the time. The performance on the foreground is different in each track, whereas the background stays always the same. The six tracks are:

1. clap: he claps hands to the rhythm of the music (no voice, only slim music version with no guitar sound and no singing)
2. resign (no voice, just gestures, still slim music version with no guitar sound and no singing);
3. guitar: he plays guitar (still no voice, reduced music version with guitar sound, but no singing);
4. sing: he sings and plays guitar (now also the voice is on: full music version with guitar sound and singing);
5. crazy: ‘violent’ performance (voice and full music version are on);
6. skeleton: he turns into a skeleton (still playing guitar and singing with full music version on).

The interaction model used in MobiLenin is based on a simple voting system. When the voting interval starts, the server triggers a voting menu in each participant’s phone, showing six choices that correspond to six tracks in the video. After 25 seconds the voting interval is over, the server counts the votes and the track receiving most votes is shown. The change of the video track results in a non-linear perception of the video on the public display. The display of the result lasts 25 seconds and then a new voting interval starts.

	Figure 2: Screenshot of the large public display of MobiLenin Copyright © Jürgen Scheible

There are several reasons why the personal mobile phone is a suitable user device for this project. First, it is ubiquitous, as practically everyone has one. Second, it allows anonymous, wireless and mobile participation in a joint social public group interaction. Third, the mobile phone provides a reliable return channel for delivering confidential user specific information back to the user, such as the winning coupon of a lottery.

The large public display serves as the main user interface for the user’s interaction. In addition to showing the music video, it indicates the start and end of a voting interval, the voting results, and notifies the audience of somebody winning in the lottery (Figure 2, above).

Experimental Evaluation
The Mobilenin system was tested in a real world setting in a campus restaurant. Qualitative as well as quantitative research data was collected with various methods. A very strong general observation was that people enjoyed using the MobiLenin system. This was expressed by laughing, happy faces, good mood, and rowdy celebrations upon winning in the lottery. This shows that the MobiLenin system succeeded in providing an engaging activity by making users feel encouraged and invited to interact with a large display, which was identified as a major challenge in the literature. Detailed information about methods used as well as the analysis of findings can be found in [4].

We can identify issues that promote the MobiLenin architecture as a platform for developing other interactive art pieces:

1. Ease of use. Easy-to-use client applications on ubiquitous personal mobile phones are used for interacting with the art piece shown on a public display.
   
   
2. Flexibility. The server as the core component of the system is controlled with simple scripts, which allow flexible tailoring of the application. Similarly, the Python scripts of the client application facilitate easy and flexible modification according to the application.
   
   
3. Modularity. Each component has well-defined interfaces and APIs. This allows for example replacing the current multi-track music video with versatile dynamic content whether it is a short movie with non-linear structure, interactive storytelling, trivia, multi-player gaming or any other multimedia art.
   
4. Real-time control of clients with private return channel. The architecture allows real-time control of the client UI for the purpose of implementing well-defined dynamic HCI patterns for both individual users and groups. The private return channel allows the application to communicate privately with individual users, which can be utilized for various purposes.

Future Work
I believe that new and exciting forms of interactive art can emerge by further developing hybrid virtual interfaces such as MobiLenin. Combining the private GUI of a personal mobile phone with the public GUI of a public display into a real-time hybrid interface fosters the social dynamics of co-located groups without distracting the user.

The MobiLenin system can be expanded in different ways by adopting different content and modifying the script of the interaction. The system is scalable so that even large groups of people can interact with the public display and the content. It would be possible to ramp up the activity of the current setup into a much more versatile interaction pattern in a more dynamic setting of a live concert. Starting from simple majority votes for the next song to be performed, the interaction could be extended all the way to complex issues such as allowing the audience to interact with the parameters of individual instruments played by the musicians on stage. This would bring me closer to my goals as an artist of giving the audience the possibility to engage in new ways in my live show.

Manhattan Story Mashup
Based on the insight gained from the MobiLenin system, I participated in co-designing the Manhattan Story Mashup. It is an urban storytelling game that combines personal mobile phones, the web and a large public display into interactive, collaborative street art. The game is based on real-time interaction between mobile and web users. In the game website’s storytelling tool, anybody is able to write stories to be illustrated in real-time by street players in New York, taking photos with camera phones.

The best illustrated stories were displayed on eleven large public displays in Times Square (Figure 3, below) and the story writers could see their illustrated own story in their web browser in real-time. The game was run in September 2006 with 184 players moving around Manhattan for 1.5 hours, shooting 3142 photos in total.

	Figure 3: Manhattan Story Mashup's public display at Times Square in New York Copyright © Jürgen Scheible

Manhattan Story Mashup [5] was organized by SensorPlanet, a Nokia-initiated cooperation on large-scale mobile device centric wireless sensor networks. A detailed description of Manhattan Story Mashup as well as an analysis of the findings of its experimental evaluation can be found in [6].

Creative Prototyping with Python on the Mobile Platform
A key factor to the successful design, implementation and deployment of both projects was the Python programming language [7], especially for the mobile phone applications. This is due to the ease of use and the fast iterative design cycle that can be applied with it. They allow focusing on developing creative ideas quickly into functional prototypes by you the artist, instead of a lengthy programming process often requiring the assistance of a professional programmer.

The rapid prototyping process deployed in the development of the mobile clients in both projects has led to a collection of many Python code snippets. Each of them serves as a building block for future interactive artworks by providing abstractions of different functionalities. These include camera, Bluetooth, SMS, graphics, keyboard keys, network access etc. The implementation of new mobile applications can be achieved very rapidly by just including the code snippets.

As part of my artistic approach when building interactive systems and interactive art pieces, I create toolkits that enable and inspire other people to build similar applications themselves. In early 2006 I made therefore my Python code snippets openly available in the Internet at http://www.mobilenin.com/pys60/menu.htm. The site also provides a comprehensive tutorial on how to learn programming with Python on the mobile platform. Since then several thousands of people from all over the world have accessed the tutorial [8] monthly. My vision is to soon have a “big garden full of beautiful flowers” out there, each representing an innovative mobile application created by inspired users and shared within the creative community. I regard my code snippets and the tutorial as a creative toolkit that stimulates user-driven innovation of new kinds of fascinating applications inspired by art. As one of the pioneers of mobile Python, I have been lately sharing my knowledge and inspiration in form of workshops at many universities and other institutions as well as companies around the world.

The Mobile Device as Art Tool
My plan is to continue contributing new code snippets, to inspire people in the creative communities to use mobile devices not just as interaction devices but also as art tools.

One such example is the mobileartblog [9] where I collect memories from my travels. A photo taken with a camera phone is fed into an image composition application built from the Python code snippets. The application creates the composite image (Figure 4, below) displayed in the blog.

	Figure 4: Mobile Art blog images created via Python application on the phone Copyright © Jürgen Scheible

This way my personal mobile phone has become my personal art tool. I use it when I get inspired by shapes, colours and forms of objects in different cities, places and situations.

I capture the moment by sending the image taken with the camera phone instantly to my blog at http://www.mobileartblog.org, transforming the image into a memorable art piece on the spot.

Conclusion
The MobiLenin system shows that a public display and personal mobile phones can be successfully combined for enjoyable multi-user social interaction, assuming sufficiently rich interactive content is available, together with a lottery mechanism to entice participation. Based on the evaluation the proposed system could offer a new form of entertainment in pubs and other public places.

Manhattan Story Mashup provided engaging experiences, triggering creativity and fostering collaboration and social interaction in form of team play. It motivated both the web and the street players to produce imaginative content, turning Manhattan Story Mashup into a collaborative street art authoring system.

The Python toolkit has been a key factor in rapid implementations of mobile applications for MobiLenin and Manhattan Story Mashup. It can also enable creative mobile phone users to become innovators by developing applications based on their own ideas - even turning the mobile device into an art tool. My message is: Be creative! Use your talent, skills, ideas and energy to inspire the world!

Acknowledgements
I wish to thank Professor Dr. Timo Ojala of University of Oulu who helped me with the text of this essay. He directed the research on MobiLenin and was also a contributor to the design and evaluation of it. I wish to thank also the co-designers of Manhattan Story Mashup Ville Tuulos of University of Helsinki and Heli Nyholm of Helsinki University of Technology as well as the Nokia support team.

References
1. H. Brignull and Y.Rogers, “Enticing People to Interact with Large Public Displays in Public Spaces”, Proceedings of Interact’03, Zurich.

2. E.F. Churchill, L. Nelson and L. Denoue, “Multimedia Fliers: Information Sharing With Digital Community Bulletin Boards”, Proceedings of Communities and Technologies 2003, (September 2003), Kluwer, pp. 97-117

3. T. Paek, M. Agrawala, S. Basu, S. Drucker, T. Kristjans-son, R. Logan, K. Toyama, A. Wilson, “Toward universal mobile interaction for shared displays”, Proceedings of CSCW’04, pp. 266-269

4. J. Scheible, T.Ojala, Mobilenin — combining a multi-track music video, personal mobile phones and a public display into multi-user interactive entertainment, MULTIMEDIA ’05: Proceedings of the 13th annual ACM international conference on Multimedia, New York, NY, U.S.A., ACM Press (2005) pp. 199–208

5. Manhattan Story Mashup website: http://www.storymashup.org

6. V. Tuulos, J. Scheible and H. Nyholm, “Combining Web, Mobile Phones and Public Displays in Large-Scale: Manhattan Story Mashup”, Proceedings of the Fifth International Conference on Pervasive Computing, Toronto, Canada, (May 2007)

7. The main website of the Python programming language, http://www.python.org

8. Python for S60 online tutorial and code snippets, http://www.mobilenin.com/pys60/menu.htm

9. The mobile art blog, http://www.mobileartblog.org

Author Biography
Jürgen Scheible is an Engineer (Telecommunications), a music and media artist. He is a doctoral student at the Media Lab, University of Art and Design, Helsinki where he runs the Mobile Hub, a prototype development environment for mobile client/server applications with a strong focus on artistic approaches and creative design. He spent several months in 2006 as a visiting scientist at MIT, Boston, CSAIL and worked previously for Nokia for 8 years. In 2006 and 2007 Jürgen was recognized as a Forum Nokia Champion for his driving vision to be a bridge builder between art, engineering and research. He is internationally active in teaching innovation workshops on rapid mobile application prototyping in academic but also professional settings e.g. at Stanford University, MIT, NTU Taiwan, Yahoo Research Berkeley, Nokia. In the 2nd half of 2007 his book “Mobile Python” will be published by Symbian Press/Wiley, bringing ‘easy programming’of mobile phones to the creative communities. He was one of the ACM Computers in Entertainment Scholarship Award winners in 2006 and Best Arts Paper Award winner at ACM Multimedia 2005 conference. His research focuses on designing multi-modal user interfaces for creating and sharing interactive artistic experiences. The philosophy behind his works is to try to bring back the depth of human feelings and emotional aspects to the digital world, which in his opinion got lost to some extent with the arrival of the fast pacing digital production technology. By inspiring others with his works he gets inspired himself. This leads to many new ideas for designing new kinds of interactive experiences for people.

Citation reference for this Leonardo Electronic Almanac Essay

MLA Style
Scheible, Jürgen. “MobiLenin and Manhattan Story Mashup - Two multimodal user interfaces for creating and sharing interactive artistic experiences.” “LEA-ACM Multimedia” Special Issue, Leonardo Electronic Almanac Vol. 15, No. 5 - 6 (2007). 10 May 2007 <http://leoalmanac.org/journal/vol_15/lea_v15_n05-06/jscheible.asp>.

APA Style
Scheible, Jürgen. (Apr. 2007) “MobiLenin and Manhattan Story Mashup - Two multimodal user interfaces for creating and sharing interactive artistic experiences,” “LEA-ACM Multimedia” Special Issue, Leonardo Electronic Almanac Vol. 15, No. 5 - 6 (2007). Retrieved 10 May 2007 from <http://leoalmanac.org/journal/vol_15/lea_v15_n05-06/jscheible.asp>.