What is mixed reality?

The term mixed reality indicates an overlap between a physical and a virtual world produced through the use of a range of more or less immersive technologies.

Diagram showing scope of mixed reality (MR) with four points on a line running horizontally. Four points are marked on the scale. Left to right these read: 1 Real Environment, 2 Augmented Reality (VR), 3 Augmented Virtuality (AV), 4 Virtual Environment. Above this scale Mixed Reality (MR) is show and encompassing a wide range of the scale from 1 to 4.

The expression was first discussed by Paul Milgram and Furnio Kishino, who proposed the idea of a ‘mixed reality continuum’, which connects what they called ‘real’ and ‘virtual’ environments, including intermediate points spanning from ‘augmented reality’ to ‘augmented virtuality’ (1994).

Mixed reality has been successfully used in a number of contexts spanning from training to medicine. Within the creative industries and the heritage sector, mixed reality has been researched extensively by Steve Benford and his colleagues at the Mixed Reality Lab at the University of Nottingham, often in collaboration with the Brighton-based performance group Blast Theory. Their pioneering use of a wide range of interfaces has been published extensively in HCI journals, and the framework derived from a decade of practice-research has been analysed in Performing Mixed Reality (Benford and Giannachi 2011).

Blast Theory member Ju Row Farr emerging from behind the rain screen, Desert Rain (1998)
Blast Theory member Ju Row Farr emerging from behind the rain screen, Desert Rain (1998), Copyright Blast Theory.

The design of the mixed reality experiences is particularly complex. This is because, as the Lab’s research has shown, mixed reality environments often occupy various points on the continuum, which could be adjacent to each other so that users could, for example, look or even pass from one to the other, as was the case in Blast Theory’s Desert Rain where the performer, in a coup de théâtre, walked through the projection of their image to hand a magnetic swipe card to the participant (1998).

Science Museum Managing Director Jonathan Newby testing Magic Leap.
Science Museum Managing Director Jonathan Newby testing Magic Leap.

Mixed reality is usually experienced through see-through head-mounted displays, whereas augmented reality tends to be experienced by pointing a hand-held display at a physical object, which then augments the object. In each case, the computer system is able to track the user’s location and augment their sphere of vision digitally. The use of the Microsoft HoloLens and the recent release of Magic Leap, the technology used in this project, have produced a lot of interest in mixed reality, indicating that we will be likely to make increasing use of mixed reality in years to come.

While museums have often used virtual reality, the use of mixed reality is still relatively new, though recent examples developed by Microsoft include the experience of a Ford car at the Petersen Locomotive Museum (2018); encounters with old Japanese artworks at Kennin-Jim, the oldest Zen temple in Japan (2018); and an encounter with an astronaut in Defying Gravity exhibition at the Smithsonian (2018). Interestingly the latter allowed four people to be in the experience at the same time, making it possible for families and friends to visit together.

In this project, we will be using Magic Leap’s head-mounted virtual retina display, which superimposes 3D computer-generated images over real world objects. The display works by projecting a field of light into users’ eyes. When testing Magic Leap, one could  literally see the physical world and, blended within it, the digital world created through Magic Leap. Interestingly, the digital world does not look like a film, but rather like sets of 3D entities, making it possible for users to establish a strong sense of presence in both the physical and digital worlds.

This project aims to create two new environments which will be designed for the Science Museum and the Natural History Museum using Magic Leap. The design of exciting content is paramount so as to ensure the experiences will be engaging and memorable. A large team of experts comprising museum researchers and curators, academics from a range of disciplines, technologists, producers, participation experts, creative content providers, sensory storytelling and education experts, among others, has been brought together for a set of creative workshops which will be the topic of the next blog post.

Leave a comment

Your email address will not be published. Required fields are marked *