VR as a Narcissistic Medium

Rebecca Rouse is a Professor at the University of Skövde (Sweden), media artist and researcher. VR is the latest medium – many claim this. But what exactly can VR do, how can we trace back its origins and what comes afterward? She coined the concept “media of attraction” and one of her latest projects is a VR experience in which the user can become immersed in a 1960s suburban Connecticut world, based on John Cheever’s iconic short story, The Swimmer.

(This interview was made within the framework of the exploratory research project rethinking intermediality in contemporary cinema: changing forms of in-betweenness, PN-IIIIDPCE-2016-0418, funded by a grant of the UEFISCDI (executive unit for financing higher education, research, development, and innovation), Romania. The interview was originally published in ACTA Universitatis Sapientiae Film and Media Studies.)

How did you start to work with mixed-reality mediums?

I have an interdisciplinary background – an undergraduate degree in Theatre and German from Brown University, an MS in Communication and Culture from York and Ryerson’s joint program, and a Ph.D. in Digital Media from Georgia Tech. I had been performing in theatre since childhood, but as an undergraduate, I became fascinated with directing and playwriting, and more specifically with technology on stage. At Brown, I directed a production of Sophie Treadwell’s play Machinal and cast a robot in the role of one of the performers. After working in local theatre and television for a while, I went back to school for a masters’ and was lucky to have an opportunity to work with Professor Caitlin Fischer at York University in her newly established AR-lab. She was incredibly generous as an advisor, and I was free to create a self-directed project in the new lab. So, I did an augmented reality version of a play for my master thesis. I translated and adapted the play Woyzeck by Georg Büchner for augmented reality, because I loved the play, but also knew there was an academic debate in theatre circles over the order of the scenes in the play. Büchner died before he finished ordering the scenes, so whenever you see it done on stage or film, it is because someone already chose the order for that production. But in my AR version, the user could choose the order of the scenes. I worked with my friend, composer Brendan Padgett, who composed the music, while I wrote the lyrics, which was greatly condensed to about half as long for the AR version – it ran about 45 minutes long. We made two casts: a cast of dancers, who we recorded in a green screen studio, and a cast of singers, who we audio recorded. The installation space for this “AR play” version of Woyzeck was created for an audience of one, who wore an optical see-through head-mounted display, as well as a hand sensor. The installation space represented Woyzeck’s town; there was a map on the floor and objects suspended in space with fishing line at different points in the room. When you touched these wooden objects, your hand entered the interaction zone (we designed hot zones around the object because we knew exactly where you were, and could very precisely track the media onto these wooden objects). We used an InterSense fusion tracking grid on the ceiling, plus head and hand sensors on the user, so the whole space had very precise tracking. Every user saw the first scene, which was a prologue introducing each character and their musical leitmotif, mapped onto a set of headless dolls (playing on the decapitation theme in the story). After that first iteration it was up to each user what they would see next: the murder of Marie first, turning the play into a murder mystery about why this woman was killed, or maybe they’d see Woyzeck committing suicide first, and then the story would be about what destroyed this man. We costumed the user in a lab coat, like a doctor. This was practical because the equipment had these battery packs that the user had to carry, and everything was wired. The user took on a few of the roles during the experience: as editor of the augmented overlay or film, they could zoom in or zoom out of the footage and shift it in terms of placement on top of physical objects; they played the part of a voyeur in the very intimate scenes; and they also became complicit in Woyzeck’s torture during a scene where you literally had to torture Woyzeck to move the plot forward. In the scene with the doctor, there was a life-sized mannequin, next to which was suspended a collapsible theatre knife, and you had to stab different parts of his (the mannequin’s) body to get that scene to progress.

I had wonderful collaborators: Michelle Moon Lee (independent programmer), Brendan Padgett (composer), Joe Pauls (filmmaker), plus advising on the technical end from Blair MacIntyre and Maribeth Gandy at the Georgia Institute of Technology. At that time at Georgia Tech they had created a program to help artists create work with AR called DART: Designer’s Augmented Reality Toolkit,  but it did not have a lot of documentation, so they were very gracious in helping us sort out how to use it.

In the end, I presented this work for a week, 50 hours in total, so about 50 people saw it. Later we made a flash version. We have a lot of documentation, video, and images online, plus all the original files. And afterward, Jay David Bolter, who was also at Georgia Tech, and had come to see the project, asked me to work for him as an assistant at his lab, which I did for about a year, after which he suggested I pursue a Ph.D. to be able to continue this type of work. I had never intended to get a Ph.D., but it was a wonderful experience, and I was lucky to have Jay as my advisor. I continued to look at technology and theatre in my dissertation and tried to develop a framework to visualize how different types of technologies impact the dramaturgy of different types of pieces and did develop a performance as a part of the Ph.D., as well as host a symposium on the theatre/ technology topic.
The performance was created in collaboration with Melissa Foulger, the Artistic Director at Georgia Tech’s DramaTech theatre, and she directed the piece, while I served as dramaturge, trying out the ideas I was developing in the dissertation. We started by co-teaching a class on integrating new technology in traditional theatre and traditional theatrical storytelling. The next semester, we put the play in production at DramaTech Theatre. In the end, we were also able to take the finished performance to a festival in Sweden. It was a play based on short stories by Haruki Murakami called After the Quake, and we used a Kinect and live projection. The play has multiple storylines and elements of magical realism: the main character, who is a writer, manipulated the projections by using gestures tracked via the Kinect. The projection sometimes represented an earthquake fissure, and at other times, a giant worm, two important elements in the piece. All along working on these theatre–media projects, I was also working on AR for cultural heritage applications, in Jay’s lab.
I continue both streams of work now. I do both AR projects for cultural heritage and theatrical work with new and old technologies, and my theoretical writing is about media history and the threads around all these things. I still work with AR, and I have three projects, on different historical topics, which are in different phases of development. Currently, I am working on my first VR project, but I’m also creating a new media theatrical experience based on a Thornton Wilder play with my students. ,

Do you see any advantages or disadvantages of being a theorist and a practitioner at the same time? Is one of your hats always more in front?

It requires different hats and roles. That might sound separated – but actually they really feed each other. If I did not make things I wouldn’t have anything interesting to write about, and if I did not stop to write about things, I wouldn’t make anything interesting.

While you were working in both of these streams and with new media, you defined a new concept, the concept of “media of attraction.” What is this concept rooted in?

I love history and working with historical material for art projects such as museums and cultural heritage sites. Since I was a kid, I have also been interested in old technologies. My dad was an engineer and he used to get a magazine, American Heritage, about the history of technology. There was one issue when I was around 13–14, about the Hollywood film actress Hedy Lamarr, but the article was not about her acting. I loved old classic films as a kid, and I could not believe that she was on the cover of my dad’s magazine with an article about her and inventing signal processing algorithms and technology for torpedoes. This really sparked my curiosity about the origins and histories of technologies. Later my Ph.D. dissertation project allowed me to look into the history of technology on stage, which was really fun. The main thing that has inspired me to develop the media of attraction concept was this feeling that we are in a very interesting time with AR. The commercialization of these technologies is happening now on a much larger level (for VR too). I have seen a couple of presentations at different conferences from researchers at the Microsoft HoloLens Lab, and the way they talk about the history of AR, and what they are doing, is really fiction – or at least, marketing. They have a commercial aim, of course, incentivizing them to claim they invented AR, but this technology has been around for 20 years. What they have done is build on the work of hundreds in the field to optimize it in an ingenious way – but that is not what they say –, they speak in a lot less nuance.

I am exaggerating a bit, but we can say that historical narratives about technology innovation get changed because of commercialization – and we are watching it happen in front of our very eyes now. I am hearing so many people at conferences saying that “we must find a language of VR,” “we must find standards,” and usually not acknowledging that those things are important because once you have standards for a language then you can have mastery, virtuosity. But not acknowledging that even when standards do get defined they are going to change again: they are temporary! So one of the chapters in the book that I am working on will be about dead conventions.

What do dead conventions actually mean?

I can give an example of older technology. In the early days of radio, which I have been researching at the General Electric archives in Schenectady, the early radio practitioners needed performers, musicians, to put on the air, but they felt that everybody was afraid of the microphone as it was an unusual piece of technology then: it captured the voice, it was uncanny  So the radio team would hide the microphone inside everyday objects. There are hilarious photographs in the archive of the recording studio where they had this elaborate floor lamp in which the microphone is hidden, and everyone is gathered around a floor lamp, performing at it. But that convention died away, and pretty quickly too, as people became comfortable with the microphone.

I have this impulse to temper some of the conversations that I’m hearing in industry and from other scholars too about “calling for a language,” “calling for conventions,” “calling for institutionalization.” I just want people to also think about the value in those early days before you had those conventions.

An interest in the early days of the film also led me to the “media of attraction” idea. I was familiar with “the cinema of attraction” scholarship, which is so fantastic from Tom Gunning, Charles Musser, André Gaudreault and Alison McMahan. They had realized that the people who worked in the very early years of film did not have a lot in common with those who worked later, as it became an industry and then Hollywood. The lack of similarity showed up not only in what they made but also in how they made it: the design practices and the goals are so different. So this group of scholars, Gunning, and others, said that the period of very early film needs its own name, “cinema of attraction.” I build on that scholarship, and what I am saying is that we can extend this way of thinking to encompass a larger range of technologies, and media of attraction can refer to any media in a very early stage, pre-institutionalization, pre-standards. Later we will see who will be a Steven Spielberg of VR, but those early pioneers, such as Georges Méliès, did not have much in common with Spielberg today. They are the ones who develop the practices that you can use today if you are working at the early stage of a new media.

We saw how actually this concept is tempering and equalizing the conversations between various actors of the new media industry. But how do you define the media of attraction concept?

Well, media of attraction can refer to any media in a very early stage, preinstitutionalization. Media at this point of development have at least four characteristics in common: they are unassimilated, meaning that there is no training program for people to go and get a degree in these subjects, and there are no formalized means of criticism. A second characteristic is an interdisciplinarity: you cannot go to school yet to major in these things, so people who work with these technologies by necessity come from all kinds of backgrounds, and this is interesting because they usually work in collaborative teams, and their practices blend and merge. The technology is still in a development stage, which means it can be very difficult to work with on a basic level, which often means a team is needed, as opposed to a single author/creator.

The third characteristic of these media is that they are participatory: they call out these people to participate more directly than they do when it becomes institutionalized and commercialized. And the last characteristic so far – I say so far because it’s true that more characteristics could emerge as I continue this work – is that media of attraction are seamed. Edges show in these early media – they are not in the mode of seamless design yet – and you can see where these seams come together at the edges in different forms of representation, and in the way that these early works are usually not self-sufficient pieces. For example, the very early film often had a live person, standing by the screen, narrating to you, so you could understand what you were looking at. This is not so different from visiting a VR arcade today, where a live person will instruct you on how to use the technology.

I hope bringing attention to early work in a new way through this media of attraction lens might shift priorities in the field slightly, perhaps helping us to value this early work more, archive these works, and mine them more thoroughly for the interesting lessons we can learn, which we’ll apply to the next medium of attraction – post-VR, whatever that might be!

You are working on both AR and VR. How do you perceive the sense of immersion conceived by these media?

I am doing both, but I have not created much artistic work in VR yet. I have made a couple of user studies creating a multi-user tracking system for a CAVE. I am just starting a project now with Marc Destefano, who is a game designer, and Brendan Padgett, who is a composer. We are working on an experimental VR adaptation of John Cheever’s short story, The Swimmer. The adaptation will be entirely sung-through (like our AR Woyzeck was) and will put you in the shoes of the protagonist, who, in this case, is a horrible narcissist. Which I feel is perfect for VR, a deeply narcissistic medium, where you are hermetically sealed in the media environment with your own reflections. I have been attracted to The Swimmer story for a long time, and now I feel this is the right medium, the right moment, for the project.

Lately, we can hear more often about how a story can fit a specific medium. Can you define why a story is good for VR?

I feel a good fit is defined by a strong dramaturgical link between the story itself, and what the technology chosen can achieve in the user or viewer. I do not think we know yet all the things that VR is “good at” – but the narcissistic quality of the medium jumps out at me for sure. Art historian Rosalind Krauss (1976) famously wrote about Vito Acconci’s early video work through this lens in her essay Video: The Aesthetics of Narcissism, in particular looking at Acconci’s work Centers from 1971, in which he films himself pointing at himself in a video monitor, as a mirror, for twenty minutes. Can we go back to the participatory characteristics of this seamed medium? Does this participatory aspect mean that it is immersing us?

I think even experiences we often categorize as “passive” can feel incredibly engaging, or immersive. If you think about a passive medium, like film, that feels effortlessly immersive – at least for me. Books, too, if they are good. So I do not think that participation is a requirement for immersion or even a guarantee. You can certainly participate in something but your mind is elsewhere, think of driving a car, but your mind wanders.

You mentioned the medium of cinema: what is separating that feeling of immersion that movies can create from mixed reality? How would you define this?

It is a different experience to do something actively and passively. In a film or novel, I feel like I surrender to the piece, but for a mixed-reality project, I am putting a bit more work in, as it is participatory, it is a collaboration. You become a part of the mechanism in different ways: you are having to work to align yourself with a tracker in such a way. In one of these new head-mounted displays, like the HoloLens, you are lining up with the physical elements, you are following directions, these mediums are not seamless. People need directions, even text, to be instructed on how to use the thing, and they need to pay a different kind of attention. If it is going well, then it might start to feel like Csíkszentmihályi’s idea of flow. I am working, I’m part of the mechanism and trying to master it. If it remains an enjoyable effort, then you might feel a kind of flow state.

So, do you think that this sense of immersion is more related to rituals and to religious feelings?

A lot of these MR practices are very performative, and from that point of view we can call these things rituals: we are taking on roles. I wouldn’t say that they are the same as the sacred ritual, but maybe they share some threads. It is also about transformation, the uncanny side of it, and the illusion of course.

What do you think theatre-makers can learn from VR creators?

VR is a spatial medium, mixed reality too, and so is theatre. Theatre makers, scenographers and writers have an old saying: “the set directs the play.” VR is a spatial medium and could learn a lot from this way of thinking. Going the other way round: what could theatre-makers learn from VR: I do not know. Disney produced an interesting VR project for the Lion King. They filmed a production from on-stage, using a 360-camera rig. This means you can be in the production, in a way. Like a fly on the wall, or in the middle of the action. You can see the performance from another perspective, see all the tricks and so forth. I love seeing how things work. But part of theatre’s joy is immediacy:
it is happening right now with other people. I guess you can have that only in multiuser VR.

Tsai Ming-liang, Taiwanese filmmaker and media artist said that for him making a VR movie was similar to doing theatre.

This is similar to the early format of Cinerama, an early widescreen format. The technology was invented in the 1940s during World War II by Fred Waller to do immersive interactive training for tank gunners. He created this system with a very wide screen to fill the field of vision, kind of like IMAX. There were speakers behind the screen, so you had directional sound, and the user sat in the gunnery turret of an actual tank: you would shoot, but with a laser or light, and they were able to calculate trajectories so you could be trained to be a better soldier. The system was credited with saving many lives. After the war, the technology was declassified, and it became commercialized as this cinema format, Cinerama, in which they got rid of the interactivity, unfortunately. To create films for it, you had to shoot with three cameras on a special rig. They did many documentary features just like at the beginning of IMAX: so in the 50s and 60s you could virtually travel to cool places or ride in a roller coaster, by viewing a Cinerama film. Then they wanted to make the leap to feature film from Hollywood, and the producers went to the directors of the famous Western movies, because of the large vistas in Westerns.

But the Cinerama technology was very difficult to work with. Hollywood actors were trained working with a single camera, and with the three-camera rig, they just did not know which camera to look at. It was even dangerous working with all this new technology. People were injured and even died because the large three-camera rig was so heavy and difficult to control. And the technology was so expensive to distribute – custom film theatres had to be constructed to show it, so it was hard to make it a large scale commercial success. A lot of this sounds like VR today: costly to distribute, because you need special equipment for your audience, and it is a new practice, you need a special camera rig to record video for it, and it’s still an open question how to best direct the audience’s attention within the 360 sphere.

How will you design the syuzhet of the VR-experience? How will you compose, how will the viewer be, will they be a ghost or an active participant?

Exactly. How do we cast the viewer? I love to think about casting the viewer or the user because of my theatre background. And then thinking about multiple roles for the user because some parts of a story might best be evoked for the user if they play one type of role, but another part of the story might be more effective if the user takes on a different role. That kind of transformation for the viewer can be intriguing, too.

What do you think about the statement that “VR is the ultimate medium?”

Well, I think this is the same statement folks will make about whatever the post-VR medium is too. So, I don’t think it’s a very meaningful statement. If you look back at early archives in film, radio, television, telephone, the internet, people always say the same things: “this technology will be democratizing,” “this technology can do anything,” you can use it for entertainment, education, for health, art… But I think this is delightful in a way, too, because you see how creative, imaginative, and optimistic people can be when they are getting to test out and play with new technology. It is exciting.

So, when people say VR is the “ultimate medium,” maybe they should add “for now.” Something will come after. It is hard to imagine what it will be because we do not know yet, but there will be something after VR, there will be a postVR medium. Someday we won’t even say “virtual reality experience.” Probably we will not have this distinction between augmented reality, virtual reality, and mixed reality. It will all come under some new, or recycled, umbrella term. Or just be unspoken, because it becomes so pervasive that it is just assumed.

 

References:

Gaudreault, Andre. 2011. Film and Attraction: from Kinemetography to Cinema. Urbana: University of Illinois Press.
Gunning, Tom. 1990. The Cinema of Attraction: Early Cinema, its Spectator and the Avant-Garde. In: Early Cinema: Space, Frame, Narrative, EDS. Thomas Elsaesser and Adam Barker, 56–62. London: BFI Publishing.
Krauss, Rosalind. 1976. Video: The Aesthetics of Narcissism. October Vol. 1: 50– 64.
Mcmahan, Alison. 2002. Alice Guy Blache: Lost Visionary of the Cinema. New York: Bloomsbury Publishing.
Musser, Charles. 1990. The Emergence of Cinema. The American Screen to 1907. Berkely: University of California Press.

 

This article was originally posted at ZipScene Magazine on October 15th, 2019 and has been reposted with permission. To read the original article, click here.