Ēkhṓs: How No Ghost and Studio Wayne McGregor are encouraging self-expression through movement with augmented reality

Over the past six months, technology experts from Digital Catapult and Niantic Labs, as well as creative leaders from Cartoon Network and Studio Wayne McGregor, have been working with some of the UK’s leading immersive studios to develop augmented reality (AR) concepts that will deliver Niantic’s vision for a ‘real-world metaverse’. The final results of Digital Catapult’s ‘Niantic Lightship accelerator programme’ were demonstrated a showcase on 2 February 2023.

Internationally renowned choreographer Wayne McGregor and Emmy award winning creative animation studio No Ghost have been working together to create Ēkhṓs.

For Studio Wayne McGregor, collaborating with Niantic and Digital Catapult through the FutureScope Lightship accelerator programme was another step towards evolving their business model from a stage-focused dance organisation to a creative company that experiments with emerging technologies, such as augmented reality.

As creative trailblazers exploring the intersection of movement and technology, Studio Wayne McGregor set a challenge for the immersive studios to solve with a Lightship powered solution. For this programme, Studio Wayne McGregor created the concept of a physical graffiti game experience that challenges everyone to make, co-create, share and interact with kinetic tags. Dance, urban sports, everyday environments, and a sense of community were to be central to the concept.

“Studio Wayne McGregor has always been fascinated by the many ways that technology can enhance our understanding of physical intelligence, and we are specifically interested in the capacity of AR to encourage movement, creativity, and expression. While other conceptions of the metaverse recreate the body in a virtual setting, we are drawn to the possibilities of AR and its ability to create a digital extension of the body in physical space. The Lightship accelerator programme has provided an incredible foundation to explore this enquiry.” – Rebecca Marshall, Executive Director, Studio Wayne McGregor

No Ghost emerged as a clear front-runner during the pitching competition during summer 2022 to develop this concept in collaboration with the Studio Wayne McGregor team. 

No Ghost is an award-winning immersive story studio from London. No Ghost has been working with immersive technologies since 2016 including virtual reality (VR), AR, installations and interactive web content. No Ghost has delivered artistic and commercial augmented reality projects, and their immersive track record received its highest accolade in December 2022, when the animated virtual reality film Madrid Noir won an Emmy.

When the No Ghost team learned that world-renowned choreographer Wayne McGregor was exploring the power of augmented reality, they seized the opportunity to join this programme. They were attracted by the ambition and scale of the brief and excited about Niantic’s Lightship platform and its capabilities for augmenting real-world map locations in particular. From early on, the team found that working with the technology that powers Pokémon Go has a halo effect: it was easier for non-technical people to envision the project when they heard it was being developed with the same technology.

Pushing the boundaries of expression with mobile phones

No Ghost’s winning concept Ēkhṓs. harnesses individual users’ self-expressions into collaborative entities that manifest above urban skylines. In spirit, their project echoes the 1970s social sculpture movement in the arts. AR technology adds a layer that brings the kinetic sculptures, named ‘Ēkhṓs.’, onto real-world map locations.

Upon firing up the Ēkhṓs demonstrator app, the user is encouraged to record expressive movements by moving their phone and themselves. This movement is translated into a beautiful ‘kinetic signature’, accompanied by a soundtrack unique to each player, championing the idea that anyone can express themselves through movement.

The No Ghost team has worked hard to make recording the kinetic signatures fun, creative, and immediate so that players are motivated to continue exploring various ways of expressing themselves. In line with Studio Wayne McGregor’s brief, they have also explored how to make the experience as accessible as possible, including accommodating neurodiverse users. 

Finding the aesthetics for collaborative, multimodal expression

The second component of the Ēkhṓs. experience is contributing to collective artworks that live in a city and will evolve based on contributions from various users of the experience. The awe-inspiring living sculptures take over the landscape in ways that would not be possible without AR.

Bringing the initial concept to life required a lot of research and development work from the team, in particular concerning how to map individual users’ signatures to the collective AR entities. Making sure that users are able to perceive the impact of their actions on the collective outcome was important – otherwise taking part in the collective experience would not feel rewarding. 

The team needed to find a balance between creative control and robust user input, and they achieved this in collaboration with Studio Wayne McGregor. The No Ghost team motion-captured the studio’s dancers’ movements using the move.ai tool. The motion-capture material provided a baseline expressive aesthetic to the ekhos, so that the user expressions would have a reference point to blend with, while the overall aesthetic remained consistent.

Speaking about the development process, the No Ghost team underlined the multidisciplinary nature of the project. For example, Ēkhṓs has a distinctive visual identity, but it is equally ambitious with its use of sound. With technologies as fundamentally visceral as augmented reality, a person’s attention often gravitates to the visual. However, Ēkhṓs  also demonstrates how sound can play a part in enhancing expressive movements in the physical world. The No Ghost team implemented a versatile system that adapts to the infinite variations users create both individually and collectively.

From a creative perspective, the team found that with AR it was challenging to find the ‘sweet spot’ in terms of the timing of when to show unfinished work to a stakeholder. In an AR project, meaningful feedback can only be collected when features are functional and properly tested and managing how soon the Studio Wayne McGregor team could expect to see tangible progress was part of the relationship-building process with the challenge owner. Nevertheless, the No Ghost team was very happy with how receptive Studio Wayne McGregor was to discuss adjustments to the direction of the concept that emerged from technical limitations, rapidly prototyped features, and so on.

Ēkhṓs all around you: harnessing the Visual Positioning System

In the past, No Ghost had worked on marker-based AR i.e. using physical objects such as QR codes in specific locations for users to trigger AR content. Ēkhṓs was different in that it required a solution that would work in several locations across a city map. Niantic’s Visual Positioning System provided the solution for manifesting the Ēkhṓs around so-called wayspots, similar to the Pokestops of Pokémon Go. Wayspots are the real-world locations Niantic supports for placing AR content as part of its Lightship platform. 

The level of ambition in the project also presented a challenge to Niantic’s platform, as No Ghost was trying to pull off user-facing features in ways that had not been tried before. The need to use third-party cloud services to store the Ēkhṓs was not something the team foresaw, and optimising the time it took for a wayspot to get localised in the user’s location, so that the Ēkhṓs would appear without delay, presented a set of development and testing tasks that took more time than anticipated. 

“It took a while to understand how the wayspots work, and how we could remotely author content – because VPS had just launched as a very new feature, there wasn’t much to be found in terms of tutorials or step-by-step guides on how other people had solved certain roadblocks “, said Lilian Hess, the Creative Director of the project.

With Niantic’s support, the team resolved the issues. Now, when an Ēkhṓs player points their phone at a wayspot location, Niantic’s platform recognises the action and that enables ‘anchoring’ of the content to the map location. This enables the Ēkhṓs to persist and evolve through multiple users’ input over time, and while in the vicinity of a wayspot, individual users can contribute their expression to the collective, living artwork. 

The team was also excited to be part of the developer community that is contributing to the coverage of wayspots, as they saw that the system becomes more powerful when more developers add to the “virtual real estate” it enables. No Ghost’s Executive Producer Luke Gibbard explains:  

“The more the map platform expands and grows is going to be a very powerful thing, because wherever you are you can say, okay, there is that billboard over there at that crossroads, and the AR content is always going to appear there for everyone. It’s virtual real estate, isn’t it, effectively. ”