Investigating Generative AI Tools for VFX…

MyWorld is a flagship programme for the UK’s creative technology sector. Led by the University of Bristol, it convenes the leading universities in the region, and over 30 technology, creative and film companies to cement the West of England’s position as a creative media powerhouse.

As part of the MyWorld programme Digital Catapult hosted and ran a 16-week acceleration program, “Catalysts and Connectors: Tools for the Creative Industries.” The program provided up to £50,000 in funding for companies in the creative industries to develop new or existing digital tools, catalysing and connecting production processes.

The participants included developers, creative studios, startups, and scaleups. With the support of Digital Catapult’s technology experts programme participants explored innovative tooling solutions, addressing industry challenges related to the creation, delivery, and assessment of experiences.

This programme was created to foster innovation-led regional growth in the West of England. Specific challenges were developed with industry partner, NVIDIA, while an Open Challenge allowed businesses to apply with relevant research and development (R&D) areas for their roadmap.

This blog explores VFX Studio Lux Aeterna’s journey through the programme. Lux Aeterna is a multi-BAFTA and Emmy award-winning independent visual effects (VFX) studio based in Bristol, working with clients across the globe to craft rich, evocative visuals for TV, film, and more.

Lux Aeterna is Investigating Generative AI Tools for VFX

Here at Lux Aeterna, we are investigating how we can integrate the cutting edge in generative AI into our creative tools and processes in a way that is innovative, practical, responsible, and that our VFX artists are excited to use.

Generative AI is artificial intelligence that uses complex algorithms and neural networks to generate new content that is similar in style and composition to the data that it has been trained on.

Over the past 18 months, we’ve seen a huge amount of development in 2D image generation AI tools with the launch of platforms such as Stable Diffusion and Midjourney. These “text-to-image” AI tools turn text prompts into complex, detailed imagery in a matter of seconds through a process called “latent diffusion”. They can also take in images and use aspects of them to influence and guide the image generation process.

We believe tools like these will change the way media is created and consumed, and could have a massive impact on the VFX industry.

That’s why we’re delighted to be one of the nine companies selected to take part in MyWorld’s Challenge Call “Catalysts & Connectors: Tools for the Creative Industries”. A project embedded within a consortium programme that’s building on the existing strength of the West of England region, exploring the future of creative technology innovation by pioneering new ideas, products and processes.

The Catalysts & Connectors programme is a 16-week accelerator run by MyWorld partners Digital Catapult, with the support of NVIDIA. The aim of the accelerator is to help companies in the creative industries develop innovative new tools to assist in the creation of exciting experiences.

Digital Catapult is the UK innovation agency for advanced digital technology, promoting new ideas and collaboration throughout industry. They are all about helping companies like Lux Aeterna reach new heights.

NVIDIA’s incredible work at the forefront of computer graphics and AI makes a lot of the R&D we’re undertaking possible, which is why we’re so pleased to have their support throughout the Challenge Call.

While 2D image generation tools have so far been used primarily to create photographic, painterly or cartoon images, there hasn’t been much focus on how these tools could be used to create images that can be used in the production of assets for VFX.

The VFX work we do requires extensive use of what are known as “maps”. Maps are 2D images that are used to represent different aspects of a scene, object or effect. Different kinds of maps are used for different purposes, such as creating 3D geometry, defining surface details, creating procedural effects, and particle simulations.

We’re exploring the development of a toolkit of text-to-image generative AI processes designed to generate maps that can be used in many different applications across our VFX work.

Here are two of the areas we’re looking into:

Generative AI for 3D Terrain Generation

Lux Aeterna is known for its work on science and nature documentaries such as Our Universe (Netflix/BBC, 2022) and 8 Days (BBC, 2019). Projects such as these often require us to create 3D landscapes, these could be the surfaces of alien planets, or Earth millions or billions of years ago. We’re interested in how we can use generative AI to generate 2D “height maps” that can be used in the creation of 3D landscapes.

Even though there are many options when it comes to creating 3D landscapes, we believe that this approach is so radically different that it could prove to be a useful alternative workflow that can leverage some of the recently established tools and processes that exist in this space.

A challenge this process presents is that these generated height maps don’t create landscapes with as much detail as we might get from other processes, such as using dedicated landscape modelling software. As such, we’re investigating how we can inject more detail and realism into our generated landscapes, as well as how we can guide and control the generation.

Generative AI for Abstract VFX

As a VFX company, we are often asked to visualise strange and sometimes impossible things. This could be interactions between particles at the quantum scale, biochemical processes within the body, or the interior of a black hole.

We employ a wide range of tools and techniques to achieve these kinds of evocative, abstract visuals, anything from procedural digital effects driven by complex mathematical formulae, to filming coloured inks and oils being dropped into a tank of water. It’s all about getting that expressive, dynamic look that can really bring those hard to imagine concepts to life.

We’re currently investigating how generative AI could be used to generate animated sequences that can be used as maps to drive these effects, creating completely new and engaging visuals in the process. The challenge of this work is to make use of a visual style that is unique to AI, but combine it with other processes to create VFX that still meet our high standards and doesn’t feel “too AI”.

This is a formative time for generative AI, and Lux Aeterna are committed to adopting responsible practices in utilising this technology. This means being aware of potential legal and ethical issues that these technologies may raise and being prepared to ask important and challenging questions.

We will be sharing more of our insights and findings as we progress through the programme.

Read more about the Challenge Call here.

Read more about Lux Aeterna’s explorations into generative AI via Lux Aeterna’s website.