“Language Is Leaving Me: A Cinematic AI Opera Of The Skin” [Part I]

Ellen Pearlman interviewed by Ivo Marais

 

Ivo: These questions are crafted to reflect the solemnity of the AI biometric cinematic opera “Language Is Leaving Me” with its themes of epigenetic trauma while at the same time inviting a deeper exploration of how it is both a personal and collective experience explored through cutting edge performative practices.

 

I: Language Is Leaving Me – An AI Cinematic Opera Of The Skin (LILM), builds on your previous work with biometrics, AI, interactivity and immersive technologies. Could you discuss the concepts behind this piece? How did your work on Noor and AIBO influence it?

Ellen: I made Noor, a brainwave opera in 2016 and AIBO, an emotionally intelligent artificial intelligence brainwave opera in 2019/2020. For AIBO I created a character with GPT-2 long before GPT Chat existed. Both operas both focused on people and events from World War II and were based on the lives of the main characters. In Noor, the main character Noor Inayat Khan was a Sufi Muslim princess spy for the allies during World War II who transmitted secret messages as a covert wireless operator in Paris back to the Allies. She was captured twice by the Gestapo and escaped twice. Because they could not break her, she was disappeared into “night and fog” and executed at Dachau. Eva, the human character in AIBO was modeled after Eva von Braun, the mistress and later wife of Adolf Hitler. The AIBO character who ran live time in the Google Cloud and modeled on Hitler was built from scratch in GPT2 before ChatGPT existed. Both operas were about World War II. I began wondering why I was making avant-garde operas about World War II when I had no experience of WWII at all. I began reading about epigenetics or inherited trauma especially of cultures of diaspora and that unlocked the puzzle to begin working on LILM and epigenetic memories, which coincided with the rise of visual AI.

Artistic Process

I: Could you walk us through your creative process? What steps did you take from the initial concept to the final piece?

E: My artistic process usually takes two to three years from concept to completion. The most difficult part is to construct a working technical proof of concept using newly emerging technologies. Often the framework for what I want to do doesn’t exist and the technology comes into being as I develop the story and performative aspects. I iterate or develop processes while doing research and locating software engineers or creative technologists to help build the piece. This includes sonic artists, coders, performers, and international organizations willing to foster and support my process.

I: What specific technologies did you use, and how did they contribute to the creation of this ‘cinematic opera of the skin’?”

E: I began experimenting with artificial intelligence cinema in 2021 for LILM during the medium’s earliest stages of development.  Through anguish and repetition, I forced myself to attain a basic understanding of  these complicated platforms and how they worked. By the next year I was able to make a seven second proof of concept video clip that was ghastly, but it worked. Around that time the AI cinema platforms that supported the tech began to stabilize and mature. I then built out the biometric and linguistic aspects.

I: As this performance involves AI, inherited traumatic memory and a focus on skin, can you explain the tactile or sensory feedback?

E: The reason I used skin or for this piece human facial muscles covered by skin is that I needed a non-invasive biometric solution to create the soundtrack that could quickly and easily be placed on a volunteer audience member. All the volunteer had to do was sit down, have the EMG patches gently placed on their face, and watch the movie. The volunteer was instructed to smile when they liked something or frown when they did not like it. I surmised that the audience and volunteer would both experience similar emotions watching the movie together for the first time.  I like to use biometrics to create a system that brings the human nervous system into a live time performance, to incorporate co-creation and a feedback loop with the audience.

Epigenetic Trauma and Creative Expression

I: Language Is Leaving Me explores trauma that affects the individual but is embedded in the genetic memory of diasporic communities. You’ve spoken about your own epigenetic memories as a non-religious Jew whose ancestors fled the pogroms of the Pale of Settlement. How did this personal connection to ancestral trauma inform the creation of this piece?

E: That’s a really interesting question. To access my own epigenetic memories, I needed to dive deeply into the formative moments from my childhood. In 2018 I was awarded an artist residency in an ‘art hotel’ in the Russian town of Peterhof, steps from the Czar’s Summer Palace.  This was before the war with Ukraine broke out. I spent the time reading about the history of genetics from Mendeley up to the new science of epigenetics. I began investigating my own family history, which I was very ignorant about beyond one generation, and during that time discovered why my grandparents had emigrated to the United States. It was because they were fleeing from the massive pogroms of 1905 throughout the Pale of Settlement in Eastern Europe. No one had ever spoken about these pogroms to me before. I just thought my grandparents came to America for the economic opportunities.

Because of my understanding of epigenetics, the strange and twisted patterns of behaviors passed down from my grandparents to my parents and by proxy to me, began to make sense. I was taken to the Polin Museum of Jewish Culture in Warsaw, Poland during my time as a Fulbright Specialist at the Polish Japanese Academy of Computer Information by my hosts. That visit to the museum really spun my head around, showing me that my ethnic diaspora was over a thousand years old. I understood that the genetic twisting of multiple traumas across millenniums was something artificial intelligence didn’t know how to deal with. Understanding that deficit and using it as a theme I worked with multilingual and multimodal artificial intelligence about my own trauma to create a visual performative opera. I discovered that AI, when confronted with the subtle memories inherent in diaspora cannot cope with it and becomes demented.

I: Trauma that lingers for over a century—such as the aftermath of the pogroms—carries a weight that transcends time. How does Language Is Leaving Me address the idea that some traumas persist, even as generations pass? Do you see your own trauma as intertwined with that of other cultures that have endured similar diasporic suffering?

E: I went through a grueling and painful process to access my own trauma. I believe my experience is probably similar to other cultures of diaspora but with different stories and characteristics. I’m talking about more than one generation because long term trauma can change the rDNA of entire populations. I’m interested in characteristics that are more internalized and nonverbal or have verbosity but not conceptuality.

I: The concept of epigenetic trauma is powerful and suggests that the past lives on in our bodies and genes. How do you feel that creative expression, particularly in this project, can help confront or process inherited trauma?

E: Creative expression can bring the free radicals or floating aspects of identity formation and behavior into focus by manipulating cultural signs and signifiers. In reference to my own epigenetic trauma, there exists certain real-world repositories of archival footage I accessed for training data. Many other cultures do not have access to archival sources. Understanding that, I took the archival and sometimes original footage and compared it to other linguistic and visual frameworks in the Chinese, Yiddish, Xhosa and Tamil languages. Because of the nature of the algorithmic transformative process, my lived experience becomes diminished or obliterated. Though that process can be viewed as a luxury it takes an enormous amount of internal psychological work, technical finesse with understanding about the structures of artificial intelligence to achieve any coherent results. Epigenetic trauma is insidious, deeply felt and encoded into the earliest aspects of identity formation. This trauma plays out generation after generation through conflict, war, aggression, genocide, hatred, and territoriality.

Epigenetic Trauma as a Medium for Immersive Interactive Performance Opera

I: In avant-garde performative tradition, there is often a tension between human emotion and technological detachment. How does the use of AI to interpret such deeply human experiences as trauma align with, or push against, this tension? What unique possibilities or limitations did this bring to the work?”

E: LILM asks “Can an AI have epigenetic or inherited traumatic memories?” The answer is no, because AI cannot interpret deeply felt human experiences like trauma. Verbal AI works by inferring vector points in space that are numerically coded using vast data resources to build its responses. When you start using visual AI the results become skewered. It might work for simple things like happy or sad facial expressions but in terms of AI’s depth to understand profound human emotions, especially cross-culturally, it exhibits dementia or intentional forgetting. This forgetting or misidentification has serious implications. The tensions in avant-garde artistic practice are to bring those underlying issues to the forefront.

I: Can you explain how visual AI is used in Language Is Leaving Me? How does it help to illustrate or interpret the concept of inherited trauma in a performance?

E: Machine learning, of which AI is a subset is used in LILM in a variety of ways. It compares dialogue, known as “text prompts” in AI.  This means I take the original English language narration of the film, run it through Google Translate and compare it with four different cursive scripts; Chinese, Yiddish, Tamil and Xhosa (South Africa) in AI using an image bank called LAION 5-B that contains over 5 billion images. Machine learning was also used during the live performance when a volunteer audience member had EMG patches put on her face. One patch was placed on her forehead measuring a negative response, or frown and the other placed on her cheek muscle to measure a smile or positive response as she watched the film. Her responses drove the soundtrack composed by a sonic artist in Touch Designer. These are the three ways that machine learning and artificial intelligence drove the performance.

When I first considered making a work about AI and epigenetic trauma, I was fortunate enough to meet Yan Le Cun, the chief AI research scientist of Meta/Facebook. I told him I was making a performance piece about epigenetic or inherited traumatic memories of cultures of diaspora. He told me that AI did not understand epigenetic trauma now, but he believed in about 20 years it would. I was stunned to find out that Meta/Facebook and affiliated researchers were already thinking about epigenetic trauma, so my initial intuition about AI’s failure with epigenetic trauma was quite accurate.

 

(End of Part I)

To read PART II of this interview, go to this link.