An Introduction to immersive UX for Digital Twins

For digital twins to reach their full potential, their interfaces must move beyond 2D paradigms into interactive immersive experiences. 

Virtual, Mixed and Augmented Reality – collectively known as immersive technologies – have grown significantly over the last decade, with significant investment and interest from large tech companies in enterprise training solutions, to the launch of consumer devices mainly used for gaming as well as room-scale immersive technologies for both consumers and professionals. The UK Digital Twin Centre is exploring the evolution of User Experience (UX) in existing and emerging applications of Immersive Technologies and how this will evolve in the near future with Digital Twins.  

What is UX 

The International Standards Organisation defines UX as “a person’s perceptions and responses that result from the use or anticipated use of a product, system, or service” (ISO 9241-210). This is the holistic journey that a user takes when interacting with a product-service-system and the combination of individual moments that affect the user while completing a given task (for example frustration, delight, or ease-of-use). It can include everything from initial perception of a product to its direct usage, and even recycling or disposal.  

UX has evolved out of the worlds of cognitive science, human-factors engineering and psychology. By understanding how our brains process the world around us, interactions and interfaces can be designed that draw on areas like sensory perception, cognition, mobility, behaviour, and social/cultural norms to create a product or service that works best for us.

Ensuring good UX takes time and effort. Useful feedback can be gained through user-focused research. The effectiveness of a given UX can be measured empirically with metrics like ‘time to task completion’ or ‘clicks-to-destination’ captured through web analytics, or the amount of thought required when interpreting an interface (known as ‘cognitive load’).  

UX is also subjective and dependent on the individual user and their context, history and mood at the time of interaction. This requires a qualitative approach through user observation, co-creation, interviews or diary studies. Using a mix of these methods iteratively improves a given UX to best fit users now and into the future. 

Why is UX important for immersive technology and digital twins? 

A digital twin is defined in part as “a virtual representation of real-world entities and processes, bidirectionally synchronized at a specified frequency and fidelity”. Each aspect of this definition has an impact on the overall UX, but here the focus is on virtual representation and the associated interactions. 

When creating a digital twin of a physical object, we often want the digital twin to follow the interaction norms that we’re already familiar with from the physical object and the world around us. This introduces both a new and old set of UX design problems. 

Flat-screen experiences have been around for more than 60 years so it’s not a surprise that most design for immersive experiences leans on the default GUI (Graphical User Interface) conventions that have been standardised by the vast userbase of modern computers, tablets and smartphones.  

Most default interfaces encountered in immersive visualisation experiences are a screen-like surface as that’s what users are most familiar with. Presented information ‘scrolls’ as on a web browser, and the user must tap and prod to select flat floating icons. Interactions created in the 1960s have carried through into immersive experiences today. This consistency reduces the barrier to entry for new users but also has the potential to restrict users from discovering new types of interactions.  

A lot of these conventions are fundamental to the human experience. From a young age we learn to interact differently between 3D objects and 2D surfaces. In recent years, we’ve become used to interacting predominantly on 2D surfaces such as smartphones or computer monitors. Deep tech such as advanced Immersive Technologies allow us to revisit the more physical, 3D forms of interaction, bringing that same intuitive learning into a digital space. Our challenge then is to discover new and richer interactions that aid the user in their use of Digital Twins. 

Reimagining interaction 

With consumer-level HMDs (Head Mounted Displays) and the emerging availability of room-scale immersive experiences, interactions with digital assets can be totally reimagined to account for volume, moving away from entirely flat-screen experiences. Available Mixed Reality technologies – such as the Apple Vision Pro and the Meta Quest 3 – enable rich experiences for viewing and interacting in 3D space.  

Combining immersive technologies with the 3D model, data captured from sensors and control mechanisms of a Digital Twin, we can create a new medium for interacting with existing assets:  

• Information overlays can be displayed beside or over the original object providing the user with useful information without requiring them to hold a tablet or laptop (Augmented Reality).  

• A scaled down model of a very large asset such as a wind turbine could be placed in the user’s environment and allow the user to perform fault analysis tasks (Mixed Reality).  

• We can go one step further and view our Digital Twin in a simulated environment. For example, if we were testing future weather conditions a wind turbine might face we could have an immersive view of the simulation as if we were stood beside our asset in the North Sea, providing detailed contextual information to the user (Virtual Reality). 

Digital Twins produce and rely on a lot of data. They also operate in tandem with other Digital Twins, producing complex Systems-of-Systems. Monitoring and controlling this new ecosystem will become too complex and noisy for existing flatscreen experiences to be effective when analysing detail, harming the overall UX. Immersive technologies can add a spatial element to helping to visualise and interact with these large, interrelated datasets simplifying this complexity for the user. 

Immersive technology plays a role with Digital Twins for designing new products as well. Crafting and sculpting in computer design software provides far greater control and detail in 3D. This creation process is helped with software and peripherals that enhance these interactions. While familiar, products like Logitech’s MR Stylus are embracing the power of immersive technology for writing and sculpting 3D creations that can quickly be grasped, moved and edited in–situ with the surrounding world – something the word processor or stylus-based experiences enabled for flatscreen writing years before. 

Prototyping tools like ShapesXR (ShapesXR — Bring ideas to life in 3D and XR) are providing users with a toolkit for easily designing room layouts or rapidly iterating on designs in Mixed Reality. This immersion while prototyping helps designers gathering feedback earlier, thereby improving the speed at which they can iterate on future designs.   

The creative industries are exploring interaction methods which utilise movement in innovative ways, creating familiar and never-before seen interactions for digital interfaces. The possibilities for usage in the digital twin space become apparent with minor adaptations. Immersive headsets are also being used more and more in 3D modelling for film and videogames, allowing the artist to sculpt parts of the environment quicker than on a screen. Exploration of immersive interactions are happening in videogames, for example the spatial interactions in Half Life: Alyx or The Climb 2. Combining this with real-world control interfaces is already being utilised for training purposes.  

Interaction and cognition 

As a building block of UX, cognitive science research into ‘embodied interaction and cognition’ tries to understand and utilise the role that action and interaction have on cognitive processes.

What might ‘thinking’ look like when unconstrained by flat surfaces? Lots of movies imagine this, like the JARVIS AI in Iron Man which works in tandem with an adaptive and exploded immersive visualisation manipulated by hand as Tony Stark thinks through an engineering problem. Almost two decades later, in 2024, JigSpace’s model viewer drew on the interactions defined in this movie to provide new ways of working with product models from jet engines to medical devices, demonstrating how fiction can help to inspire the design of new interaction paradigms. 

The wider context of emerging immersive UX  

However, there are still challenges with reaching a UX suitable for a workforce working with Digital Twins and immersive technology. Currently HMD’s do not have the same accessibility features that a flatscreen experience is required to implement. All members of a given workforce would therefore not be able to use these devices and so be unable to realise the benefits. Accessibility of these devices moves beyond the expected constraints into the wider workplace context. Some challenges to consider with this current generation of HMDs use are:  

• Can the user wear a headset? 
• Can the user wear a headset? 
• Is the user colourblind?  
• Does the user get motion sick? 
• Can the user perform the needed movements for interaction? 
• How does wearing the headset impact their hair or make-up?  
• Does the headset get heavy or hot over time?  
• Can the user wear it with glasses?  
• How does the user feel using the device with an audience around them?  
• Does the audience feel as if they are being recorded?
• Does the audience understand when to communicate with the headset wearer?  

As immersive experiences continue to improve and become more commonplace in a business environment these challenges are expected to be overcome. One such area of innovation focuses on the question: what does an immersive experience that can recognise and adapt to user constraints and actions at a given time look like? And how does research into the balance of text vs graphics in data visualisations inform better immersive information communication and decision making?

We’re seeing the beginnings of technology that can do this already with the combination of HMDs with contextual AI features. Combining this with a Digital Twin has the potential to enrich the overall UX by providing context-specific information to users as and when they need it. By combining this with Digital Twins, detailed insights can be provided to the user and give them deeper interaction and control of the world around them. 

An example of this might look like a maintenance engineer being called to fix an aircraft, their HMD would recognise the component they are looking at and gather data from the Digital Twin to explain what and why the fault had occurred, all provided to the engineer with minimal need for them to interact. The engineer could then decide on the correct fix or improvement to reduce the chances of the fault happening again.  

What’s next? 

Immersive UX is a fast-moving, complex, and multi-faceted topic and this blog has aimed to introduce why UX is an important consideration for the practical deployment of digital twins into industry. The UK Digital Twin Centre is exploring how to make digital twins more accessible and meaningful – a key aspect of this is making the user experience the best it can be, to make the digital twin experience fully come to life.