An Introduction to immersive UX for Digital Twins

Virtual, Mixed and Augmented Reality – collectively known as immersive technologies – have grown significantly over the last decade, with investment and interest from large tech companies funnelled into the metaverse and enterprise training solutions, and into consumer devices mainly used for gaming as well as room-scale immersive technologies for both consumers and professionals. The UK Digital Twin Centre is exploring the evolution of User Experience (UX) in existing and emerging applications of immersive technologies and how this will evolve in the near future with digital twins

What is UX 

The International Standards Organisation defines UX as “a person’s perceptions and responses that result from the use or anticipated use of a product, system, or service” (ISO 9241-210). This is the holistic journey that a user takes when interacting with a product-service-system and the combination of individual moments that affect the user while completing a given task (for example, frustration, delight, or ease-of-use). It can include everything from initial perception of a product to its direct usage, and even recycling or disposal.  

UX has evolved out of the worlds of cognitive science, human-factors engineering and psychology. By understanding how our brains process the world around us, interactions and interfaces can be designed that draw on areas like sensory perception, cognition, mobility, behaviour, and social/cultural norms to create a product or service that works best for us.

Ensuring good UX takes time and effort. Useful feedback can be gained through user-focused research4 where UX can be measured empirically with metrics like ‘time to task completion’, ‘clicks-to-destination’, as well as through web analytics or the amount of thought required when interpreting an interface (known as ‘cognitive load’).  

UX is subjective and dependent on the individual user and their context, history and mood at the time of interaction. This requires a qualitative approach through user observation, co-creation, interviews or diary studies. Using a mix of these methods iteratively improves UX to best fit users now and into the future.

Why is UX important for immersive technology and digital twins? 

A digital twin is defined in part as “a virtual representation of real-world entities and processes, synchronised at a specified frequency and fidelity”.5 Each aspect of this definition has an impact on the overall UX, but here the focus is on virtual representation and interactions. 

When creating a digital twin of a physical object, we often want the digital twin to follow the interaction norms that we’re already familiar with from the physical object and the world around us. This introduces both a new and old set of UX design problems. 

Flat-screen experiences have been around for more than 60 years so it’s not a surprise that most design for immersive experiences leans on the default GUI (Graphical User Interface) conventions that have been standardised by the vast userbase of modern computers, tablets and smartphones.  

Most default interfaces encountered in immersive visualisation experiences are a screen-like surface as that’s what users are most familiar with. Presented information ‘scrolls’ as on a web browser, and the user must tap and prod to select flat floating icons.7  

There are examples of adaptations of CAD (Computer Aided Design) models easily shifting from desktop modelling software into immersive viewers. This consistency reduces the barrier to entry for new users but also has the potential to restrict users from discovering new types of interactions.  

A lot of these conventions are fundamental to the human experience. From a young age we learn to interact differently with 3D objects and 2D surfaces. Deep tech, such as immersive technologies, provides the opportunity to revisit physical forms of interaction bringing that same intuitive learning into a digital space.

Reimagining interaction 

With consumer-level HMDs (Head Mounted Displays) and the emerging availability of room-scale immersive experiences, interactions with digital spaces can be totally reimagined to account for volume, moving away from entirely flat-screen experiences. While familiar, products like Logitech’s MR Stylus are embracing the power of immersive technology for writing and sculpting 3D creations that can quickly be grasped, moved and edited in situ with the surrounding world – something the word processor or stylus-based experiences enabled for flatscreen writing years before. 

The combination of technologies that constitute mixed reality devices – from the Apple Vision Pro and the Meta Quest 3 – enable users to quickly view models of chairs, shoes or aeroplane thrusters, providing a completely new way of experiencing contextual feedback. Prototyping tools like ShapesXR are providing users with a toolkit for easily designing room layout or rapidly iterating on immersive mixed reality designs.  

The creative industries are exploring interaction methods which utilise movement in innovative ways, creating familiar and never-before seen interactions for digital interfaces. The possibilities for usage in the digital twin space become apparent with minor adaptations. 

Interaction and cognition 

As a building block of UX, cognitive science research into ‘embodied interaction and cognition’ tries to understand and utilise the role that action and interaction have on cognitive processes. 

What might ‘thinking’ look like when unconstrained by flat surfaces? Lots of movies imagine this, like the JARVIS AI in Iron Man which works in tandem with an adaptive and exploded immersive visualisation manipulated by hand as Tony Stark thinks through an engineering problem. Almost two decades later, in 2024, JigSpace’s model viewer drew on the interactions defined in this movie to provide new ways of working with product models from jet engines to medical devices, demonstrating how fiction can help to inspire the design of new, real-life interaction paradigms.

Other work capitalises on the sudden access to a user’s context and surroundings to craft meaning in a location familiar to them. The ‘Method of Loci’ or ‘Memory Palace’, draws on our powerful spatial cognition processes to improve memory and recall by encoding information with imagined points around a room or building. Some AI systems use something not dissimilar for encoding high dimensional data into a 3D representation where distance between items is a salient datapoint – called embeddings. Is there a natural synergy here to cognitive processes that can be explored with digital twins in the near future? 

The wider context of immersive UX 

Immersive UX extends far beyond the screen or headset, and there are key design questions that need to be carefully considered before successfully involving the user in product-service-systems. Some of these questions include:  

• Does the user need to wear a headset?   

• How does this impact hair or make-up?  

• Does the headset get heavy or hot over time?  

• Can the user wear it with glasses?  

• How does the user feel using the device with an audience around them?  

• What are the audience’s thoughts?  

• Does the audience feel as if they are being recorded?  

• Does the audience understand when to communicate with the headset wearer?  

As immersive experience continue to improve and become more commonplace in a business environment, what does an immersive experience that can recognise and adapt to actions at a given time look like? And how does research into the balance of text vs graphics in data visualisations inform better immersive information communication and decision making? 

Digital twins comprise several advanced technologies working in combination and UX will play a vital role in orchestrating these technologies with the user at the centre.  

What’s next? 

Immersive UX is a fast-moving, complex, and multi-faceted topic and this blog has aimed to introduce why UX is an important consideration for the practical deployment of digital twins into industry. The UK Digital Twin Centre is exploring how to make digital twins more accessible and meaningful – a key aspect of this is making the use experience the best it can be, to make the digital twin experience fully come to life.