VR and Theater Finally Make a Climate Baby: “From Sea To Rising Sea” Interview with Creator-Director-Choreographer Mary John Frank

Technology and the theater are closer together than ever, and our hope for the planet may depend on it. 

After my seventh time seeing Chekhov brought to life through NFTs, robots, and an electric blue cherry orchard at the Baryshnikov Arts Center this summer (Arlekin Players Theater’s “The Orchard,” starring Jessica Hecht and Mikhail Baryshnikov, headed to Boston in early November), I couldn’t get over the idea that the keys to our future maybe forged in an unusual alliance of technology and theater. 

That notion was solidified this week, when I strapped on a VR headset, grabbed a couple of controllers, and traveled through a unique VR musical experience that blends Broadway, synchronized swimming, climate change, technology, and education: “From Sea to Rising Sea.”

The ominously title piece is a six minute interactive VR/360 underwater adventure featuring Broadway star Phoenix Best, artistic swimmers The Aqualilies, a marine biologist, and the closest thing to actually scuba diving you can feel without fear of Jaws’s jaws. 

The timing is fitting, because it’s Climate Week in NYC, and following showings at UN World Oceans Day, Pioneer Works, and Earth Day, you too can catch up with this immersive and excellently produced experiment.

In FSTRS: Plot wise – the characters highlight the importance of voting, regenerative ocean farming, wind farms, and blue carbon. The film ends with tangible action items like where to donate to kelp farmers or coral restoration efforts, or where to find a climate-focused job, and of course the reminder to vote. 

Beyond the actionable and empowering messages, what I took away from the experience (made in partnership with  Oceanic Global and youth group, Zero Hour) was a celebration of the beauty of the natural world we live, move, work, and make theater in. 

Noteworthy is the proliferation of climate messages targeted to youth – who are, as Whitney Houston, our future – and perhaps our last hope for action. From Sea to Rising Sea, with its musical bent and uplifting tone is in good company with another recent mainstream production that has its characters go out into nature to find clarity and truth: Shakespeare’s “greenworld” was perfectly on display with Shaina Taub’s fun, youth-friendly adaptation of As You Like It in the Park this year (Public Works by The Public Theater).

To understand the journey of creating this original, complex immersive project, I caught up with the project’s lead creator, director and choreographer, Mary John Frank. See our full interview below on the process of marrying art and technology (especially during COVID), Mary John’s take on the role of art and activism, and which Broadway musical she thinks would make the best VR/360 immersive experience. 

While technology has enabled and emboldened the destruction of the planet, it’s heartening and perhaps necessary to see that it might also play the role of savior. Perhaps the fault lies not in our stars (or oceans, or machines), but in ourselves. 

BOTTOM LINE: 

You can catch  FROM SEA TO RISING SEA, a VR musical experience that follows a flock of synchronized swimmers and a marine biologist as they guide a concerned citizen on an underwater adventure, as it exhibits this week during Climate Week NYC this Friday September 23rd, 2022, from 3-11 pm at:

SVA Gallery Chelsea
601 West 26th Street
15th Floor, Suite 1505, New York, NY 10001

Tickets: https://www.eventbrite.com/e/the-2022-marketplace-of-the-future-tickets-344994005357

It’s 6 minutes, kid-friendly and adult-compelling, and I can confirm that my normally sensitive stomach had no nausea from the VR setup, thanks to thoughtful filmmaking and the short duration. 

You also can watch it on your phone (without a headset). 

And catch the trailer for free, here: https://vimeo.com/581171549

INTERVIEW

Fareeda Ahmed: You have worked mostly as a choreographer and director. What led you to Virtual Reality? 

Mary John Frank: I was working on an immersive dance performance in 2015/16 and was looking for interesting ways to film it. Flat screen coverage didn’t seem to truly capture the 360 nature of the live show, and while researching the best ways to film the experience, I learned about 360 filmmaking and VR. From there I applied to some grants in this space and got lucky. I received a grant from Harmony Labs and Google DayDream to make my first 360 film. 

FA: I’m coming at this from a theater bent, so: What exactly is 360 film? Is it the same as VR? What category does “From Sea To Rising Sea” fall under?

MJF: The version of FROM SEA TO RISING SEA that you just experienced in the Quest 2 was a 360 film with 3 Degrees of Freedom (3DoF) meaning you could look all around, to the left and right, up and down, and you could also pivot. Many VR experiences, “full VR,” have 6 Degrees of Freedom (6DoF) giving the viewer more opportunities to move through the environment and to interact with objects. With 6DoF, you can also teleport to different places in a scene and move more freely within the space. FROM SEA TO RISING SEA falls under the 3DoF/360 category with a little bit of interaction.  

FA: How did you learn to make work in this medium? 

MJF: I learned through experimentation and also by attending a VR filmmaking residency at Almanack Screenwriters where technical directors and VR producers guided a few filmmakers, including me, through the production process. Creators from theater and film who had made 360 and VR films discussed their experiences, then we wrote scripts that later became 360 films. We explored how storytelling techniques from live theater and dance were useful in VR e.g. how to direct a viewer’s eye in a frameless universe. 

FA: Talk to me about the concept? What was the inspiration for this project?

MJF: Ocean-based solutions to the climate crisis, smart scientists, and the multitude of climate action items available to us were the inspirations. I wanted to point Viewers to the helpers, creative thinkers and optimists who are dedicating their lives to healing and rebuilding our planet with climate justice as a guiding principle. We have seen a lot of content that focuses on the problems associated with climate change. I wanted to bring awareness to some of the solutions, and also tell a story with characters who are willing to change, and are trying to help despite the grim statistics.

FA: Why VR & 360 filmmaking? What does technology add to this project?

MJF: Immersion and fun. I wanted to show off the beauty of the planet as part of the call to action. This 360 experience gives viewers the opportunity to float in a kelp forest in the Pacific, hover over the glittering sea, and visit serene national parks like the Channel Islands, virtually. We also take viewers inside a synchronized swimming formation and onto a  boat, where they can learn about thermal expansion and erosion, while also having the option to look around and take in the surrounding coastal landscapes. 

We hope viewing this 360 film will offer a nice reprieve, a little escape into nature, and a reminder of why this gorgeous planet is worth saving. 

FA: Art and activism aren’t strangers. How does technology enhance or change this relationship? 

MJF: In this project, VR technology helps to bridge the gaps that exist between the audience, a work of art / film, and an important cause. Software like Unity allowed us to create very simple interactions at the end of the experience that connect audiences to climate actions, organizations, and policies that are helping us to reach net zero carbon emissions. 

FA: Talk to me about the process of finding collaborators? Are you a technical person?

MJF: I was introduced to our partners, Oceanic Global, and several of our technical collaborators through an art and tech incubator in New York, called New Inc., and through VR communities like Artizen and Unity. I began reaching out to advisors, potential collaborators, and funders in 2018, and we eventually filmed in 2020. More on our talented team HERE. 

And, no, I would not call myself a technical person. I am, however, very curious about new technology, specifically how AR & VR will enhance, expand, or alter mediums like dance. Overtime, I have learned how to collaborate effectively with smart technicians who know more about web development and coding than I do. On this project I worked with Occurrent Arts: https://www.occurrentarts.com, Dot Dot Studios https://dotdot.studio, and Sam Baumel https://www.sambaumel.com.

FA: What kinds of stories and experiences are suited to VR, and which aren’t? 

MJF: In narrative experiences, stories that unfold around the viewer (think immersive plays or dance experiences) work well. Also stories that allow viewers the freedom to look where they want to look and explore while still being connected/guided to a clear theme or message. Some experiences I have enjoyed are “Dear Angelica,” Cirque Du Soleil’s, “O,” and “Wolves In The Walls.” I am drawn to these projects because of how they utilize the 360 space, the quality of the VFX and/or animation, and the very private and imaginative spaces they’ve created. 

Regarding what might not work, experiences with quick cutting or motion that isn’t stabilized can be jarring in VR especially with live action storytelling. Sensitivity to and mindfulness of the viewer are key, as motion sickness is real and there is a deeper level of embodiment for audiences in this medium since they are wearing a headset and are fully immersed in a world vs. looking at a screen. Also stories that are frame-dependent, or scenes where a director is banking on everyone looking at one moment in order to follow the story or move the story forward throughout the piece, will likely not be as impactful. 

Jessica Brillhart, has written some fantastic pieces about VR storytelling that are helpful to read when getting started in this medium: 

https://medium.com/the-language-of-vr/how-to-greet-a-rebel-unlocking-the-storyteller-in-vr-d40b2cc05f55
https://medium.com/the-language-of-vr/virtual-reality-s-fundamental-question-ee94763a917d”>https://medium.com/the-language-of-vr/virtual-reality-s-fundamental-question-ee94763a917d

FA: What is your hope/dream for this project? What are some of your goals for the work?

MJF: Impact is our number one goal.  If we inspire a few people to show up to the polls in November and vote with climate top of mind I will feel we have done our job. I also hope the project encourages Viewers/Students to dream on the type of planet they want to live on and that they (and all of us) will take steps to make those dreams a reality. 

Re: other goals, so far we’ve shared the project in classrooms with students, exhibited in a museum, licensed the film to VR libraries thanks to SIMA Academy and Tech Row, built a 360 app, and premiered at a UN event. Getting to exhibit during Climate Week is exciting and I’m currently hoping for a seamless and successful exhibit later this week!

FA: What kind of reception is the piece getting? Any interesting reactions? 

MJF: Many of our Viewers have said “thank you for making this ”after viewing, especially young climate activists and scientists. Below is my favorite response we received from a Ph.D. Student focusing on Environmental Energy Policy who generously typed up his thoughts: 

I believe education is the only way to make an impactful and sustaining change. And one can only learn when the means of education is effective and powerful. This short VR video does exactly that. Thanks for the effort. The world needs more of FROM SEA TO RISING SEA. – Dane Kim (Ph.D. Student, Michigan Technological University)

Students have also created climate drawings, poems, dances and skits after watching the film, like the below:

FA: How does someone view this piece without a headset? How difficult were those adjustments to make?

MJF: We’ve built a 360 app for our in-person exhibits so that people who don’t want to put on a headset, given the times / pandemic, can watch the piece on their phones. Because we wanted to offer Viewers some tangible action items, lists, and other ways to get involved we needed to create a customized app vs. using a pre-built 360 app like on YouTube or Vimeo.

The biggest challenge was not the interactive elements, but getting the 360 film to play seamlessly across mobile browsers and devices in the customized app. This wasn’t hard for our developer, Kate Stevenson, to do, but required time and patience. We had to make tweaks to the sound, the film’s resolution, the subtitles, and other elements like this. Any time there are updates to Safari, Browsers, iPhones etc. we find glitches, so have to test regularly. 

FA: What projects are next for you? 

MJF: I am developing an Augmented Reality dance experience, where AR technology will partner with live dancers. The underlying theme is an exploration of “tech for good” and, currently, the audience will have an opportunity to determine how the experience ends through their devices. I’ve also just directed my first commercial, and I’m hoping to do more of that kind of work this year as well.

FA: If VR were a lunch table, who would be sitting there? 

MJF: Five years ago I think this table would have been occupied by technicians, gamers, and forward thinking cool kids. My hope is that the VR lunch table continues to become roomy and inclusive with lots of tables, stools and chairs, pushed together, occupied by creators and technicians from a variety of art forms, races, genders and ages.

FA: If you could take any Broadway musical and turn it into a VR film, which one would you do and why? 

MJF: “Wicked” would be my first choice. I love the music and the potential that exists in this story for flying and floating in VR. Moreover, the “Wizard Of Oz” is my favorite film, and I love the message and story of female friendship. “Defying Gravity” is such a powerful song, and I think it could be even more so in an immersive, spatialized context. 

FA: Also, how has this project changed you as an artist?

MJF: Thank you for asking this! I feel like a more seasoned director and filmmaker after making this. I learned to have more patience and to let go of perfectionism during this experience. There was a lot that I didn’t know about this medium when I started, especially regarding the underwater filming components so I had to ask for help, frequently. While filming in nature and in remote areas, we didn’t have a monitor given the 360 equipment we had access to, so I had to have a lot of faith in the cast and our DP. The project became incredibly collaborative and I hope to bring the lessons learned on this production into the next one.