Super Epic Happy Fun Time

As a technical director at Pixar, I had many different stints, in many different departments, on many different projects. Because my responsibilities were so varied, people would always ask me what I was working on. I would joke that I was working on a “Make Movie” button, and that soon my entire job would be automated.

Well now, with the rise of AI, I decided that maybe it finally was time to build the “Make Movie” button.

Predating the rise of text-to-video models such as Sora, this project utilized the AI tools freely available at the time.

Over the course of this project I not only learned about the technology, but I also learned how critical the human touch still is in the age of AI.

Execution

Each episode starts as a.json file which defines the name of the episode, the location, the background audio track, the characters in the episode, each of their lines, and the camera position for each of their lines.

• Then a script would distribute all of the tasks on Deadline and the entire episode would render.
• Generate audio for each line
• Generate face animation using Rhubarb for the character
• Render each clip in Houdini OpenGL
• Run Stable Diffusion XL on each frame
• Combine all the frames together and add music using FFMPEG

Make Movie ▶️

Setup

The design aesthetic of the world resembles the prolific art of Keith Haring because his characters lack teeth, fingers, and the other details AI particularly struggles with.

The show is staged as a multi-cam sitcom which allowed me to set the camera positions ahead of time in each scene.

The show is in Japanese because the text-to-speech models at the time sounded uncanny when speaking English. I figured a lot of people are used to watching Japanese anime and I could just subtitle it.

Results

This project was more than just a race to making infinite video content. The limitations of the technology served as creative inspiration. The constraints forced me to make creative choices that resulted in a unique and intentionally imperfect result.

Although the YouTube viewership was limited, the project did strike a chord with a specific “super fan”. Because of his enthusiasm and interest in the show, I featured him in one of the episodes.

AI may be able to accelerate the animation process, but the human connection is still the most critical component.