Imagine/: Artificial Intelligence Comes to Theatrical Design

Lately, I have been seeing examples of Jason Jamerson’s theatrical design work on Instagram – images he has made in collaboration with Artificial Intelligence apps.

According to a recent article in The New York Times, “[A.I.] apps like DALL-E 2 and Midjourney are built by scraping millions of images from the open web, then teaching algorithms to recognize patterns and relationships in those images and generate new ones in the same style.”

Jason reached out to me by email several days ago with this message:

Hello Michael,

I’ve seen you around Instagram quite a bit, and I think we know a couple of the same people.

I have been working with the recent A.I. networks such as DALL-E 2 and Midjourney to create concept art for theatre designs. You can see a bunch of examples here.

It’s easy to give it a try. If you haven’t looked into it, you sign up at www.midjourney.com and that will direct you to a chat program which you may or may not already know called Discord. On Discord you can “chat” with Midjourney, and it will reply with images based on your text prompt.

I am wondering if you would be interested in having a conversation about this as a design practice?

I was interested! The following is the result of our conversation.

Michael Schweikardt: Here are some anecdotal thoughts on the design work I have been seeing on Instagram (and I write this before trying any of the A.I. technology myself) – the images are stunning. They possess a sophistication of taste that surprises me. From a designer’s perspective, the use of composition, color, line, shape, texture, and light are all exquisite. “Impressive” is the word I keep returning to.

What excites me: As an instructor, I am particularly interested in helping students materialize their design ideas into scenery for performance. The defining attribute of scenic designers is their ability to transform meaning into material environment, but this moment of transformation often seems mystical, and elusive. This A.I. technology might be used as a tool to make this moment of transformation less mysterious.

What gives me pause: In my view, scenic designs create spaces that are meant to facilitate action in real time. In other words, set designs are engines for action. What happens in space over time is as important (if not more important) than how the space looks. Script analysis and dramaturgy are the keys to approaching set design from this point of view. Set design, without script analysis and dramaturgy, is decorating. When it comes to A.I. theatrical design work, my questions are: where is the dramaturgy; where is the script analysis; how do the designs facilitate action; how do the designs relate to the written drama; is this merely decoration? Does A.I. bring scenic design back to pre-Robert Edmond Jones time – not in the sense that scene design tends towards Realism, but in the sense that set design becomes something that performers act in front of, as opposed to inhabiting?

Jason Jamerson: I have seen some designers get angry about A.I. scenic designs that I posted on Instagram. But every one of us checks Google Images for inspiration. Now, Midjourney can look at every picture on Google Images and collage a new image out of them for us. It can do it a hundred times, and we can pick the absolute best and most compelling versions to use as our inspiration. The ones on my Instagram are just a demonstration, and I’m glad they’ve stirred up a lot of talk.

Set design isn’t just making something beautiful on a stage. The real art is in the sculpting of the negative space that the performance occupies. A.I. doesn’t understand that (yet). But it is definitely giving us an I.V. drip of pure, unfiltered visual impact. Up until now, we’d read the script, look at a bunch of research, and venture alone along a dark road to imagine a synthesis of our needs and passions for a design. Then one day we’d turn up at a meeting with sketches, ground plans, etc. Now, for the first time, thanks to A.I. networks, we can bring something with us on that road where our ideas melt together. I think A.I. is a lantern. As long as we’re in charge of where we’re going.

MS: I  agree with you that designers use Google Images for inspiration – I use it, I guide students and young designers in how to use it – and I can see how the A.I. programs can be thought of as an interesting evolution of that practice.

Did you happen to see the NYTimes article about Jason M. Allen of Pueblo West, CO who won a prize in an art competition for an artwork he made using Midjourney? It is a good source for linking to other research and writing on the topic of A.I. artmaking, but I found it a little sensational; a little “hair on fire”, if you will. Although there are a lot of ethical questions at play that interest me, at the moment, I am not super hung up on worrying about the death of the artist.  My concern (as I expressed above) is more specifically about how A.I. work with theatre making.

JJ: I did see that article. I think we are living in an in-between-time. Probably, rules for art competitions will immediately change to ban A.I. art, because human skill – the artists’ craft in communicating their intent through their hands – is implied in an art competition. But for commercial art – for book covers, or stock photography where the cheapest, most beautiful image wins – I think we’re going to see a huge upheaval, and probably jobs lost.

But, my interest is in people who love to create, people who have no intention of giving up designing. Can’t we think of A.I. as just another tool in our tool belts?

MS:  This is where I start to get excited. How can we adopt A.I. as part of a larger process?

JJ: The tricky thing that we can’t quite understand is what exactly does the neural network [A.I.] “see” when it consumes a billion images as its training data. It can’t understand what a stage design is, but it does recognize trends. When every production photo it encounters features an open area, with levels, planes, and entrances / exits, it sees that. And when every scenic design for The Wizard of Oz has a yellow brick road, it sees that, too. But, it also sees incorrect things. If you do a Google Image search for “Oklahoma musical” you’ll get an overwhelming number of show posters. And if you put that same phrase into Midjourney, chances are, you’re going to get a set design that has the word “Oklahoma” in a mangled font somewhere on it. Midjourney doesn’t know that the poster isn’t the show itself.

MS: That explains why, when I asked Midjourney to create images for “Oliver musical”, I received image after image with the word “Oliver” in them.

JJ: Yes, but, for the most part, there’s an incredibly positive trait in Midjourney’s work. You see, the entirety of picture-taking-humanity has spent countless hours culling photos and posting the good ones to the internet, to the point that Midjourney is spot on with what we like to see – and it gives that back to us in a completely new image.

If you look at the “Jackson Pollock A Streetcar Named Desire” from my Instagram, you’ll find everything you need to stage the show. A Streetcar Named Desire is a great example, because it requires a scenic designer to remake it, each time, in their own unique voice. But, you also have to have the bathroom door, the curtain dividing the room, the front door that you can see both sides of, the daybed, the kitchen, the staircase up to Eunice’s apartment, etc. And for the most part, in the Midjourney created images, it’s all there. Midjourney isn’t just making a textural and distressed version of New Orleans for performers to stand in front of. Midjourney learns the physical needs and action of the play because it’s seen five thousand production photos of A Streetcar Named Desire, and it combines them with surprising, compelling decor. I don’t see the images that Midjourney comes up with as merely decoration. If you ask Midjourney for a set design, I think you’re often going to get one.

While Midjourney is successful at conceiving a stage design for a play, it is also completely incapable of devising a new solution to a script’s challenges; Midjourney is limited to what has appeared in existing production photos (what it can find on the internet). But, we don’t want the A.I. to design the set. We just want it to give us new, interesting concept overlays. We want to remain the designers.

I am wholly satisfied with my ability to come up with a set design for a play that supports and empowers the director and the actors, creates a mood for the piece, and, when appropriate, provides a broader metaphorical visual frame for the show. So, what do I need Midjourney for? When I am in pursuit of those magical moments that come from the surprising combination of two ideas, or when a script problem and a novel visual approach harmoniously come together – when it all just clicks – well, that strange, heartbreaking synthesis shows up in Midjourney’s images quite frequently.

If I input the right text as the prompt.

Midjourney still relies on me to choose what matters. In fact, some of the best results come not from inputting “set design for Romeo and Juliet” but from asking Midjourney to whip up a “Renaissance Italian palazzo made of skulls and roses”.

MS: Can you tell me more about how you are using prompts – what works, what doesn’t, and why?

JJ: I will try! It’s a bit mystical. Midjourney can accept a lot of data, and will try to include most of it. Things toward the front of the prompt are given more weight, but it will accept the whole paragraph and try to spit something out. I find what works best is giving Midjourney a clear style, and a subject with descriptors. The prompt “Ansel Adams photograph of Homer Simpson’s House” probably yields decent results.

Here’s a sample prompt that I used for one of my Instagram posts:

“theatrical costume designs for the tempest in the style of hokusai :: shakespeare :: costume design :: theatre design, stage design, costume design, fashion, realistic, 4k, octane render, lighting design, highly detailed”.

You can explore how different ideas and styles separated by “::” affect the prompts.

MS: How does “::” work?

JJ: A comma is a soft separation of ideas; it tells Midjourney to combine these elements. A double-colon is a hard separation; it tells Midjourney to include both items. You can also give added weight to a term. If you type “/imagine elephants::10 ::monkeys:: in a jungle” you’re saying that elephants are ten times as important as anything else and must be included in the result (10 being the max). Here’s a bunch of other parameters you can use: https://midjourney.gitbook.io/docs/imagine-parameters.

MS: We’ve talked a lot about Midjourney. What about other A.I. networks like DALL-E 2?

JJ: There are about a dozen neural networks out right now, and Midjourney is the most compelling one, because it frequently defaults to a sketchy, webby kind of visual style. It just inherently creates concept art. It’s really beautiful. The reverse of that is that the others, like DALL-E 2 and Stable Diffusion, make more realistic things, but they tend to be way less compelling. So, in my prompts for Midjourney, I spend a lot of time trying out terms like “HD, 8k, realistic, highly detailed, symmetrical face” to force some realism into the results. But, if you also take into consideration that Midjourney is very good with style, then you can use “styles” like “ArtStation”, or “Octane Render” (a highly realistic rendering engine used in Hollywood), or “DeviantArt”, or “yearbook photo” to get a style that promotes realism, or concept art, or anime, or whatever.

MS: I am curious to experiment with how this technology might help the process of materializing a design idea for a production. At the moment, I think the fundamental question that emerges is: what are the possibilities for A.I. (neural) networks as part of an overall design practice?

JJ: Right. Ultimately, we don’t want to ask Midjourney to give us a scenic design (even if it does that better than we expect). Instead, we should see ourselves as farmers, or harvesters, of Midjourney’s fruits. From way back in the day when we collaged cut-out magazine pictures and photo copies from research books, to the age of Google Image searches, we have been looking for tiny little sparks of inspiration to put in our basket. I see Midjourney as a hundred times more potent than that, but still related to that part of the process.

Jason Jamerson is a scenic and production designer whose work has been seen on stage and film in New York and across the country for nearly twenty years. He is a Professor of Virtual Production and Immersive Media at Louisiana State University, where he holds a joint appointment in the School of Art and the School of Theatre, teaches classes related to production design, animation, game design, VFX, and digital asset creation; works collaboratively in the LED volume Virtual Film Studio, and pursues research related to computer simulations of digital environments for media and design practice. He is passionate about developing artists who explore authentic, driven storytelling and the rapidly changing toolset we use to create it, including digital technology, animation, game engines, virtual film studios, and A.I. His work can be found at www.instagram.com/jasonjamerson

Michael Schweikardt enjoys a successful career as a set designer working for opera and theatre companies across the United States and abroad. He is Assistant Professor of Set Design at San Francisco State University’s School of Theatre and Dance. Essays in Teaching Critical Performance Theory in Today’s Theatre Classroom, Studio, and Communities (Routledge 2020), Text and Presentation, 2019 (McFarland 2020), and Performing Ethos demonstrate his research focus on the tension between materiality and ephemeral performances. He serves as managing editor for design at The Theatre Times and is co-creator and co-editor of Prompt: A Journal of Theatre Theory, Practice, and Teaching. His work can be viewed on his website: www.msportfolio.com