From presence to co-presence: ‘being there’ in virtual and mixed reality

The term ‘presence’ is a contraction for ‘telepresence,’ which has been described as the ‘defining experience of VR’ (Steuer 1992). Conventionally, the functioning of ‘presence’ in virtual environments indicates the degree to which participants feel that they are somewhere other than where they ‘physically’ are while experiencing a computer-generated simulation (Sheridan, 1992a and 1992b; Barfield and Weghorst 1993; Slater and Usoh 1994; Barfield, Sheridan, Zeltzer, and Slater 1995). It follows that the concept of ‘presence’ in virtual reality is not so much concerned with ‘aura’ or awareness of ‘self’ or ‘other,’ but rather with ‘the illusion of being here or there’, which, as Frank Biocca indicates, suggests that ‘the fundamental issue at the root of the problem of presence in virtual reality is the perception or illusion of reality (2001: 550, original emphasis). So, if presence defines the illusion of being in virtual reality, what is the relationship between presence and immersion, a term nowadays often used to define virtual and mixed reality experiences?

While presence may be linked to immersion, it is important to note that presence and immersion do not coincide. Mel Slater and Sylvia Wilbur describe immersion in a virtual environment as a quantifiable aspect of a display technology, indicating the features of the technology employed to immerse the participant, while presence refers to ‘a state of consciousness, the (psychological) sense of being in the virtual environment’ (Slater and Wilbur 1997: 604f). It has been argued that the higher the level of immersion, the higher the level of presence, and ‘the more that a system delivers displays (in all sensory modalities) and tracking that preserves fidelity in relation to their equivalent real-world sensory modalities, the more that it is “immersive”’ (Slater 2003). However, for Slater, the experience of presence is ‘a human reaction to immersion’ which means that, given the same level of immersion, participants may still experience presence in different ways (Ibid.). Moreover, it is known that ‘it is not necessary for the users to feel completely immersed in order to perceive presence’ suggesting that ‘low immersive technology can create high presence’ (Seichter in Wang et al 2009: 48). So, if immersion is quantifiable and presence isn’t, why not just measure the former? The answer to this lies in what are described breaks in presence which occur when ‘the participant stops responding to the virtual stream and instead responds to the real sensory stream’ (Slater, Brogni and Steed 2003). A break in presence may occur due to ‘uncanny valley’ a phenomenon that refers to ‘a sense of unease and discomfort’ experienced by some participants when engaging with realistic virtual humans (Tomlinson 2000). So, the lack of presence indicates that participants in a virtual environment did not engage with the illusion of the virtual reality. This in turn shows that an understanding of presence can lead to important findings for the design of virtual reality experiences.

There is evidence that presence may not only be a response to an environment but may also be produced in response to mediations generated by artifacts, both physical and conceptual, ‘between actors and between them and objects both near and remote’ (Mantovani and Riva 1999: 541). This proposition constitutes an understanding of presence that is ‘relational and interactive’ (Ibid.). What is particularly interesting in this context, is that virtual reality environments in which presence is experienced can consequently be described as ‘networks in which people and things construct themselves mutually’ (Ibid.). Such networks suggest that a sense of ‘presence’ may therefore be a response to behaviours and relationships that arise within an ecology in which the actor, or participant, defines and co-constructs, often with others, their active place within the illusion provided by the virtual reality.

Presence is a key measurement not only for virtual but also for mixed reality environments. In this context, it is especially useful to capture users’ sense of presence through a range of categories, including spatial presence, social presence and object presence (Witmer and Singer 1998). Presence, which has been associated to both objective and physical and subjective and psychological components (Slater and Steed 2000), is often assessed through a range of methods, including questionnaires and physiological data. Some questionnaires have been widely discussed and used as a point of inspiration by others (Usoh et al 2000 and Witmer and Singer 1998). Mel Slater’s is particularly useful as it utilises a range of parameters that are suitable in a performative context, namely the subjective sense of ‘being there’, the extent to which users react to the environment as if ‘real’ and the sense of having visited a ‘place’ rather than looked at images representing it (Slater 1999). Critics pointed out that presence questionnaires filled out directly in the virtual reality are less likely to cause a break in presence (Schwind, Knierim, Haas, Henze 2019: 1), than those conducted, as is conventional, immediately after the experience (Slater and Steed 2000; Slater 2004; Sanchez-Vives and Slater 2005).

By comparison with virtual environments, mixed reality environments present a complexity that affects users’ sense of presence within them. The notion of mixed reality was introduced by Milgram and Kishino (1994) and subsequently elaborated by Steve Benford et al who argued that mixed realities tend to be composed of multiple displays and adjacent spaces (Benford, Greenhalgh, Reynard, Brown and Koleva 1998; Benford and Giannachi 2011). One of the most common factors affecting presence in mixed reality is the co-habitation of physical and simulated elements, and the transitions from one to the other. Another is the presence of multiple visitors. When reflecting about presence in mixed reality, versus presence in virtual reality, it is therefore important to note a shift towards ‘social action, interaction and construction of meaning’, as multiple and often ‘interacting users’ inhabit environments with material objects engaging a range of senses (Wagner, Jacussi, Broll, Kuutti 2009: 249).

Just as the experience of presence in virtual reality has been described as being a function of focus (Fontaine 1992), selective attention (Triesman 1963), vividness of an experience and the level of interaction (Sheridan 1992a, 1992b; Steuer 1992) or involvement (Witmer and Singer 1998), the experience of presence in mixed reality is even more so challenged by the fact that it is constructed not in one but multiple adjacent and often fluid spaces, and not only by one but usually by multiple visitors that may well be co-operating but also may be distracting each other during a given experience. Social presence, the feeling of being with another person and presence, the feeling of being in a place, brought together have been described as producing co-presence (Ijsselstein and Riva 2003), and it is co-presence that is a very important parameter for the understanding not only what users experience in mixed reality (Wagner, Jacussi, Broll, Kuutti, 2009) but also how they co-operate in continuing to play along with the illusion generated through it.

With museums increasingly planning for augmented and mixed reality experiences and Magic Leap, our technology provider, planning to launch Enterprise Suite and Device Manager, aimed at ‘bringing physical and digital worlds together as one’ to allow people to ‘either physically present or digitally co-present to collaborate’ (Magic Leap 2019), it is crucial to gain further insight into how people not only experience but also actively collaborate to produce the mixed reality. Through this project we will be able to build on the existing literature on presence and co-presence in virtual and mixed reality environments and reflect on the role of performance within this context.

References

  • Barfield, W. and Weghorst, S. (1993) ‘The sense of presence within virtual environments: a conceptual framework’ in G. Salvendy and M. Smith (eds) (1993) Human Computer Interaction: Software and Hardware Interfaces, Atlanta Georgia: Elsevier Publisher, 699–704.
  • Barfield, W., Sheridan, T., Zeltzer, D. and Slater, M. (1995) ‘Presence and performance within virtual environments’, in W. Barfield and T. Furness III (eds) (1995) Virtual Environments and Advanced Interface Design, Oxford: Oxford University Press, 473–513.
  • Benford, S., Greenhalgh, C., Reynard, G., Brown, C., & Koleva, B. (1998) ‘Understanding and constructing shared spaces with mixed reality boundaries’ ACM Transactions on Computer-Human Interaction, 5(3), 185–223.
  • Benford, S., and Giannachi, G. (2011) Performing Mixed Reality, Cambridge, Mass.: The MIT Press.
  • Biocca, F. (2001) ‘Inserting the Presence of Mind into a Philosophy of Presence: A Response to Sheridan and Mantovani and Riva’, Presence: Teleoperators and Virtual Environments, 10:5, 546-56.
  • Fontaine, G. (1992) ‘The experience of a sense of presence in intercultural and international encounters’, Presence: Teleoperators and Virtual Environments, 1:4, 482-90.
  • Ijsselstein, W., & Riva, G. (2003) ‘Being there: The experience of presence in mediated environments’, In G. Riva, F. Davide, & W. A. Ijsselstein (Eds.), Being there: Concepts, effects and measurements of user presence in synthetic environments, Amsterdam: IOS Press, pp. 3-16.
  • Magic Leap (2019) ‘Spatial Computing for Enterprise: rewrite the rules of your industry’, https://www.magicleap.com/news/news/spatial-computing-for-enterprise-rewrite-the-rules-of-your-industry.
  • Mantovani, G. and Riva, G (1999) ‘Real’ Presence: How Different Ontologies Generate Different Criteria for Presence, Telepresence, and Virtual Presence, Presence: Teleoperators and Virtual Environments, 8:5, 540–50.
  • Milgram, P. and Kishino, F. (1994) ‘A Taxonomy of Mixed Reality Visual Displays’, IEICE Transactions on Information Systems, E77-D12, 449-455.
  • Sanchez-Vives, M. and Slater, M. (2005) ‘From presence to consciousness through virtual reality’, Nature reviews. Neuroscience 6, 4, 332-339.
  • Sheridan, T. B. (1992a) ‘Musings on telepresence and virtual presence’, Presence: Teleoperators and Virtual Environments, 1:1, 120–5.
  • Sheridan, T. B. (1992b) ‘Defining our terms’, Presence: Teleoperators and Virtual Environments, 1:2, 272–4.
  • Schwindt, V., Knierim, P., Haas, N., Henze, N. (2019) ‘Using Presence Questionnaires in Virtual Reality’, CHI 2019, May 4-9, Glasgow, Scotland, UK.
  • Slater, M., and Usoh, M. (1994) ‘Representations systems, perceptual position, and presence in immersive virtual environments’, Presence: Teleoperators and Virtual Environments, 2:3, 221–33.
  • Slater, M., & Wilbur, S. (1997) ‘A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments’, Presence: Teleoperators and Virtual Environments, 6(6), 603-616.
  • Slater, M. (1999) ‘Measuring Presence: a Response to the Witmer and Singer Questionnaire’, Presence: Teleoperators and Virtual Environments, 8:5, 560-566.
  • Slater, M. and Steed, A. (2000) ‘A virtual presence counter’, Presence: Teleoperators and Virtual Environments, 413-434.
  • Slater, M. (2003) ‘A note on presence terminology’, Presence Connect, 3. Available online: http://www0.cs.ucl.ac.uk/research/vr/Projects/Presencia/ConsortiumPublications/ucl_cs_papers/presence-terminology.htm.
  • Slater, M., Brogni, A. and Steed, A. (2003) ‘Physiological Responses to Breaks in Presence: A Pilot Study’, Presence 2003, the 6th Annual International Workshop on Presence. Available online: https://www.researchgate.net/publication/2893422_Physiological_Responses_to_Breaks_in_Presence_A_Pilot_Study.
  • Slater, M. (2004) ‘How colorful was your day? Why questionnaires cannot assess presence in virtual environments’, Presence: Teleoperators and Virtual Environments, 13: 4, 484-493.
  • Steuer, J. (1992) ‘Defining virtual reality: Dimensions determining telepresence’ Journal of Communications, 42, 73–93.
  • Tomlinson, B. (2000) ‘Dead Technology’, Style, 33, 316-335.
  • Triesman, A.M. (1963) ‘Verbal cues, language and meaning in selective attention’, American Journal of Psychology,77, 206-219.
  • Usoh, M., Catena, E., Arman, S., Slater, M. 2000) ‘Using Presence Questionnaires in reality’, Presence, Teleoperators and Virtual Environments, 9:5, 497-503.
  • Wagner, I., Jacucci, G., Broll, W., Kuutti, K. (2009) ‘On the role of presence in mixed reality’, Presence Teleoperators and Virtual Environments, 18:4, 249-276.
  • Wang, G.X. And Schnabel, M.A. (2009) Mixed Reality in Architecture, Design and Construction, Milton Keynes: Springer.
  • Witmer, B.J. and Singer, M.J. (1998) ‘Measuring Presence in Virtual Environments: A Presence Questionnaire’, Presence, Teleoperators and Virtual Environments, 7:3, 225-240.

Placing mixed reality in the museum

One of the main challenges of this project is to create two experiences taking place in two augmented worlds located within two museums that have very distinct spatial characteristics. Each experience is to be formed by a temporary installation comprising a set of rooms with specific architectural and scenographic qualities. This means that while the setting of each experience will feel a bit like an augmented set design, or possibly even an escape room, or a complex museum installation, it will, at the same time, form part of the broader visiting experience of each of the two world-leading museums.

To research how to build these spaces, and to understand what place they would occupy within the two museums, the audience research teams in the museums carefully considered exactly how visitors will be arriving at the two spaces, and what they will be encountering before and after each of the experiences. Museum visiting is, of course, part of a spatial, architectural experience, in which, as shown by architect Bernard Tschumi, ‘the very heterogeneity of the definition of – space, action, and movement – makes it into an event, that place of shock, or that place of the invention of ourselves’ (Ibid.: 258, added emphasis). The very architecture of the museum too, then, should be a place of action, of self discovery, and social cohesion.

Project team in Magic Leap headsets undertaking walkthrough of proposed experience

Each of the spaces we aim to build could be described as a world within a world that is transformed, on various occasions, into yet another world. Each of the spaces will in fact be located within one of the museums, and form part of their complex architecture, yet also be augmented through the use of the mixed reality.

To understand how visitors would move within such a complex space, and so have a better sense of the conditions under which the event would be produced, the team conducted a number of walk-throughs, testing how participants, who were team members on this occasion, would behave in narrow spaces when wearing a headset; how they might relate to each other; how easily they would understand and respond to instructions; what the timing of the experience might be; and how they would cope with being asked to take on different roles and acts as onlookers, participants, gamers, learners, teams, or even researchers at different points in time within the same experience.

Project team in Magic Leap headsets undertaking walkthrough of proposed experience

As each experience is structured around a specific use of space and time, it might be interesting to remember that the two terms are in fact strongly inter-related and constitute the fundamental concepts through which we define orientation. Human actions occur in time and through space, and, according to the Oxford English Dictionary, space denotes time or duration, and time is in fact a space or extent of time.

Each of the experiences entails the construction of a time present (the fictional here and now in which the visitor is placed), a chronological time (the narrative order of the events), and plot time (the inter-relation between the time present and the chronological time of the fiction), as well as a performance time (marking the duration of the overall experience). Spatially too the experiences will be located within the museums, and, within that, in a number of rooms, which will on occasion appear augmented.

Project team in Magic Leap headsets undertaking walkthrough of proposed experience

The complex hybrid space created by the overlap of these physical, fictional and digital spatio-temporal narrative structures are best described by using Katherine Hayles notion of ‘enfolded spaces’ in which what is witnessed ‘is no longer a homogeneous context for a given spatial area, but rather pockets of different contexts in it’ (in De Souza e Silva 2006: 269). The idea of a folding space cannot but bring back Gilles Deleuze’s theorisation of the fold, as an in-betweenness of spaces, able to represent dialectical opposites such as organic and inorganic, inside and outside (1993: 13). This folding spatio-temporal environment, which can no longer easily be represented in a two-dimensional drawing, or as a chronology, or even a conventional architecture, as it is only live as a mixed reality, constitutes a new kind of place within the museum.

References

  • De Souza e Silva, A. (2006) ‘From cyber to hybrid: mobile technologies as interfaces of hybrid spaces’, Space and Culture, 9:3, 261-278.
  • Deleuze, G. (1993) The Fold, tr. B. Conley, London and New York: Continuum.
  • Tschumi, B. (1996) Architecture and Disjunction, Cambridge, Mass.: MIT Press.

Prototyping at Factory 42

For the Factory 42 development team, the greatest feature of using Magic Leap – the element that pushes it ahead of any other augmented or mixed reality technology to date – is its ability to combine the physical and digital worlds together. The simple ability for digital content to react to physical objects in the real space: a table, the floor, even the player’s own hands, is a revolution in realism and interactivity. This concept is continuously at the heart of our designs and the prototypes we are developing.

For a few months now, the development team have been exploring the various functions and features of the Magic Leap One devices. Since then, we’ve put together prototypes and demos that test the breadth of Magic Leap’s features, including image tracking, hand and gesture recognition, eye tracking, telemetry and world reconstruction to name a few. We’ve even built our own online multiplayer scenes in which players can all wear Magic Leap headsets and share the same experience together.

Presented in this post are a handful of some of the features that are available with Magic Leap.

World Reconstruction

In order for our digital content to interact with the world around us, the Magic Leap device needs to scan its surroundings and build up a mesh of the environment; this is called World Reconstruction.

Once Magic Leap has created a digital version of the world we can experience true mixed reality.

Magic Leap One user's view of scanning of environment with mesh overlaid on office

Magic Leap Create app scanning environment in app setup
Create by Magic Leap

Image Tracking

The ability to recognise a specific image, and follow its position around the room, is a feature common to many mobile AR apps. Image tracking can be a useful way to activate your digital content at key locations in your scene; they can make pictures come to life.

View through Magic Leap headset showing tracking of an image on a desk

Important design notes to consider when using Image Tracking:

  • Image Tracking typically works best if the image is stationary and of a decent size; we don’t often go smaller than A5 if we can help it.
  • When deciding on which pictures to use, bear in mind that the Magic Leap image tracker discards all colour information, so try to choose images with high contrast.
  • Glossy or reflective finishes will interfere with detection of your image.
  • Tracking works best when the image is flat. Granted Magic Leap does not need to detect the entire image for recognition to occur, but bending or distorting your image around a 3D surface will increase the likelihood of errors.

Hand Recognition and Gestures

Hand Tracking, similar to Image Tracking, is the recognition of key points on the user’s hands. In addition to tracking all five fingers on both hands, Magic Leap is able to detect a number of specific hand gestures made by the player. These include: Open Hand, Closed Fist, Thumbs Up, Pointing, L shape, C shape, Pinching and OK.

Drawings of the eight hand gestures tracked by the Magic Leap One headset
Source: Magic Leap Creators Portal (June 2019)

View through Magic Leap One headset showing handtracking overlaid on user's hand

 

Accompanying visitors at the Natural History Museum

To assess what future audiences might enjoy as part of museum visiting, this project started by looking into existing studies exploring how current audiences experience museums and heritage organisations.

Two initial workshops were held at the Natural History Museum and the Science Museum, during which teams from the museums illustrated how they conduct audience research and researchers from the University of Exeter presented on business modelling based on the visitor experience, and the use of ethnomethodological-oriented methods in other projects (e.g. the EU-funded VISTA AR project and the AHRC-funded Performance at Tate project).

It is common practice in museums, galleries, heritage organisations, and in many other experience-based organisations, to use ethnographic methods to study audience behaviour.

Visitors and exhibits in Hintze Hall, Natural History Museum, London, including a blue whale skeleton is suspended from the ceiling.

Ethnographic methods are also widely used in the study of mixed reality. For example, the Mixed Reality Lab at the University of Nottingham has devised a series of ethnomethodologically-informed methods, including a technomethodology [PDF], which are aimed at assisting researchers at the Lab to understand the challenges in the design of specific human computer interactions.

It therefore seemed that the project could start by conducting an assessment of the current visitor experience at the Natural History Museum by using accompanied visiting.

Accompanied visits use some ethnographic strategies to observe what visitors do in museum, galleries or other heritage organisations. During an accompanied visit, a member of staff accompanies one or more visitors, either observing them or engaging more directly with them, while annotating spontaneous behaviour and conversations. It is known that accompanied visits are a great way to give voice to visitors and are particularly useful when studying groups from under-represented backgrounds (Haywood 2018).

On this occasion the museum audience research team had asked project members to look into what the museum experience was like for families and adults under 35, what people did, whether there were moments of confusion or obstacles and barriers, and to reflect about what conclusions and lessons could be learnt from this experience that could be applied to the project more broadly.

Visitor at Natural History Museum, London, viewing text and information panel on human sex cells.

I accompanied a couple, a man and a woman, in their late twenties. They had both visited the museum as children but had not returned for a number of years.

They first headed to the human biology exhibition where I first noted that they did not take photos, though they did buy a souvenir.

I did not have permission to photograph the two visitors, but had permission to capture generic visitor behaviour, and found this was usually consistent with the behaviour of the two visitors I accompanied.

Visitors at Natural History Museum, London, including information panels and two dinosaur skull exhibits.

It was noticeable that visitors often struggled to get close to the exhibits due to the large number of people in each room, which often resulted in moving more quickly than anticipated to another exhibit, or even a different room.

Visitors at Natural History Museum, London, viewing marine vertebrae exhibits.

One of the visitors I accompanied was interested in seeing the marine vertebrate gallery and looking at the dinosaurs, and so both visitors spent a long time in the marine vertebrae exhibition, reading the interpretation carefully, often discussing specific artefacts with each other.

Visitors at Natural History Museum, London, viewing and photographing T.rex exhibit.

Once they reached the roaring T. rex animation, they stopped, observing that the exhibit offers a photo moment, but then they walked quickly through the remaining dinosaur galleries as they had to leave to for a meeting.

Visitors at Natural History Museum, London, viewing exhibits.

Other groups, accompanied by other project members, showed some interest in sensory experiences, and younger people especially liked touching exhibits.

As Dani Parr found by following her group, audiences ‘expect to learn every time they come to the museum’. In fact, for all groups, there was an expectation to learn, to see something new, and a great excitement about interactive and responsive exhibits.

Exhibits at the Natural History Museum, London, including blue whale and rhinoceros.

The only two ‘wows’ in my group occurred, respectively, in the Hintze Hall, when the whale was first spotted, and when I explained that the aim of the project was to experience a dinosaur in mixed reality.

Reference:

Naomi Haywood (2018) ‘Accompanied Visits as a Tool to Understand Visitors’ Experiences: A Critical Reflection and Proposed Typology’, Visitor Studies, 21:1, 135-147, DOI: 10.1080/10645578.2018.1503876

Creative workshops

Workshop whiteboard with the word audiences written on it
Understanding future audiences was one of the drivers of the creative workshops held at the Science Museum and the Natural History Museum.

Since our project started a few months ago, various teams, each representing an indispensable area of expertise, have been researching what audiences of the future might wish to engage with and how best to facilitate this engagement. Crucial questions have been asked including, for example, how to provide audiences with valuable learning experiences in mixed reality; how to iteratively test with audiences; how to capture what individuals and groups might experience in a mixed reality environment; and how to ensure the widest possible reach and accessibility of our work.

Bethan Ross, Senior Audience Researcher at the Science Museum Group introducing the team to audience profiling practices at the Science Museum.
Bethan Ross, Senior Audience Researcher at the Science Museum Group introducing the team to audience profiling practices at the Science Museum.

The project started with a series of creative workshops, which were held at the Natural History Museum and the Science Museum, and facilitated by the immersive and content production studio Factory 42.

Documentation of the first creative workshop at the Science Museum.

Documentation of the first creative workshop at the Science Museum.

The first two workshops were informative and served the purpose of introducing the various teams involved in the project to existing research. It allowed us to explore the multi-sensory possibilities of storytelling in Magic Leap; brainstorm possible wow factors; and identify possible challenges by conducting multiple risk assessments. The workshops were extensively documented to create a rich resource for those who might be interested in researching the work in years to come.

The Science Museum creative workshop lead, Dani Parr, overlooking the team’s exploration of what they thought a robot could look like.

These two initial workshops were followed by a further set of creative workshops in each of the museums. The first day was about imagining what worlds could be built. The second day consisted of a writing workshop and was led by an expert team of creative storytelling writers. The third day focused on working through some of the stories, paying particular attention to the technological challenges, and included a workshop with museum staff looking at a number of stories, and the fourth day was about storyboarding the three strongest options, focusing on branching narratives and audience agency. Finally, during the last day two stories were selected so that they could be shared and discussed with museum staff and the wider team. The feedback indicated at this point that both stories should be kept ‘alive’ to test Magic Leap opportunities before making the final selection.

Documentation of the brainstorming workshop about robots at the Science Museum.

Documentation of the brainstorming workshop about robots at the Science Museum.

Both creative workshops were designed and led by the experienced theatre director and participation producer Dani Parr from the Almeida, who involved participants in thinking about which kinds of robots or dinosaurs we might expect to see in Magic Leap; how groups would work together in mixed reality; how the chosen space within the museum would work in practice; what active learning could mean in this context; how the experience could be accessible and give a sense of exploration while also being semi-scaffolded and orchestrated on the day; how the story might be extended before and after the museum visit; and what would be the relationship between fact and fiction, imagination and museum learning.

Discussions focused on how the design of the experience would need to appeal to people from different age groups, different cultures, different competences; work for families, friends, and individuals; have multi-sensory triggers; be accessible and scalable for touring; encourage repeat visits; have a wow factor and be scientifically accurate.

What is mixed reality?

The term mixed reality indicates an overlap between a physical and a virtual world produced through the use of a range of more or less immersive technologies.

Diagram showing scope of mixed reality (MR) with four points on a line running horizontally. Four points are marked on the scale. Left to right these read: 1 Real Environment, 2 Augmented Reality (VR), 3 Augmented Virtuality (AV), 4 Virtual Environment. Above this scale Mixed Reality (MR) is show and encompassing a wide range of the scale from 1 to 4.

The expression was first discussed by Paul Milgram and Furnio Kishino, who proposed the idea of a ‘mixed reality continuum’, which connects what they called ‘real’ and ‘virtual’ environments, including intermediate points spanning from ‘augmented reality’ to ‘augmented virtuality’ (1994).

Mixed reality has been successfully used in a number of contexts spanning from training to medicine. Within the creative industries and the heritage sector, mixed reality has been researched extensively by Steve Benford and his colleagues at the Mixed Reality Lab at the University of Nottingham, often in collaboration with the Brighton-based performance group Blast Theory. Their pioneering use of a wide range of interfaces has been published extensively in HCI journals, and the framework derived from a decade of practice-research has been analysed in Performing Mixed Reality (Benford and Giannachi 2011).

Blast Theory member Ju Row Farr emerging from behind the rain screen, Desert Rain (1998)
Blast Theory member Ju Row Farr emerging from behind the rain screen, Desert Rain (1998), Copyright Blast Theory.

The design of the mixed reality experiences is particularly complex. This is because, as the Lab’s research has shown, mixed reality environments often occupy various points on the continuum, which could be adjacent to each other so that users could, for example, look or even pass from one to the other, as was the case in Blast Theory’s Desert Rain where the performer, in a coup de théâtre, walked through the projection of their image to hand a magnetic swipe card to the participant (1998).

Science Museum Managing Director Jonathan Newby testing Magic Leap.
Science Museum Managing Director Jonathan Newby testing Magic Leap.

Mixed reality is usually experienced through see-through head-mounted displays, whereas augmented reality tends to be experienced by pointing a hand-held display at a physical object, which then augments the object. In each case, the computer system is able to track the user’s location and augment their sphere of vision digitally. The use of the Microsoft HoloLens and the recent release of Magic Leap, the technology used in this project, have produced a lot of interest in mixed reality, indicating that we will be likely to make increasing use of mixed reality in years to come.

While museums have often used virtual reality, the use of mixed reality is still relatively new, though recent examples developed by Microsoft include the experience of a Ford car at the Petersen Locomotive Museum (2018); encounters with old Japanese artworks at Kennin-Jim, the oldest Zen temple in Japan (2018); and an encounter with an astronaut in Defying Gravity exhibition at the Smithsonian (2018). Interestingly the latter allowed four people to be in the experience at the same time, making it possible for families and friends to visit together.

In this project, we will be using Magic Leap’s head-mounted virtual retina display, which superimposes 3D computer-generated images over real world objects. The display works by projecting a field of light into users’ eyes. When testing Magic Leap, one could  literally see the physical world and, blended within it, the digital world created through Magic Leap. Interestingly, the digital world does not look like a film, but rather like sets of 3D entities, making it possible for users to establish a strong sense of presence in both the physical and digital worlds.

This project aims to create two new environments which will be designed for the Science Museum and the Natural History Museum using Magic Leap. The design of exciting content is paramount so as to ensure the experiences will be engaging and memorable. A large team of experts comprising museum researchers and curators, academics from a range of disciplines, technologists, producers, participation experts, creative content providers, sensory storytelling and education experts, among others, has been brought together for a set of creative workshops which will be the topic of the next blog post.

Thoughts on mixed reality at the Science Museum

The Science Museum has a long, long history of interactive exhibits and use of digital technology. The earliest explorations of how hands-on experiences could bring science and technology to life date back to the 1930s.

Image of 1930s interactive gallery
Schoolboys in the Children’s Gallery of the Science Museum, March 1934

Interactivity is now a fundamental part of the museum’s offer and making sure that these exhibits are audience centred and fun is something we take incredibly seriously.

Interior image of Wonderlab: The Equinor Gallery at the Science Museum
Wonderlab: The Equinor Gallery at the Science Museum

The museum also adopted digital experiences early and remains a leader in this field. In recent years the museum has begun to explore the opportunities afforded by immersive technology to bring objects to life—to explain how they work, their historical context and the lives and experiences of people who used them. So, far we’ve explored Tim Peake’s return to Earth from the International Space Station in the Soyuz descent module in VR…

… and how VR can explore the complex mathematics behind the design of the Handley Page aircraft which is the centrepiece of Mathematics: The Winton Gallery.

Over the past ten months, driven by the Audience of the Future: Demonstrators application process, we’ve had dozens of conversations exploring the exciting affordances of mixed reality in the museum. When colleagues have asked us what we’re creating we’ve had to tell them that it is about robots and that the experience will combine: museum exhibit, digital environment, adventure game, immersive theatre, and escape room. So, here are our early thoughts and questions which are shaping our thinking:

1. Interactive experience

The two VR pieces that the museum has produced to date are linear. The experience unfolds and the audience member watches and listens; they are fundamentally passive. The opportunity for our new experience to offer agency to the audiences is incredibly exciting as we know that interactivity increases audience enjoyment and learning.

However, interactivity presents the project challenges too:

  • To what extent is the experience a sandbox environment that audiences can explore and to what extent is the experience on rails?
  • What does this mean for the experience design?
  • How do we inboard audiences to an interactive experience which will for almost all be their first use of mixed reality? Can this be part of the experience’s narrative?
  • How do we ensure the experience is inclusive and accessible?

2. Social experience

For most audiences, visiting the museum is a social experience. People come in groups. Whereas VR is an isolating, solitary experience, mixed reality has the potential to create a shared experience in which a group of visitors can collaborate or experience something together. This is hugely exciting for us as we can realise all the immersive benefits of VR but with this critical social element.

Again, there are challenges here too:

  • What if the group know each other well?
  • What if they don’t know each other?
  • What if they don’t speak the same language?
  • What if someone undertakes the experience alone?
  • Could the experience work for groups of school children?
  • If one or more member of the group can’t use the mixed reality headset for whatever reason, how can they still be part of the experience?
  • Are there opportunities for “Instagram moments” that can be shared on social media with friends and family beyond the walls of the museum?

3. Story-based experience

Stories are an effective way of connecting visitors with the museum’s displays. The way the museum traditionally approaches stories—called “narratives” internally—is centred around objects from the museum’s collection and the stories they tell, delivered through architecture, exhibit design, interpretation and programming. Our approach with this project will be completely different. Here we are beginning with the experience design and the story. There will be no physical objects—though there will probably be some 3D scans.

This approach really flips the museum’s traditional ways of thinking and working on their head. Conversations about experience design, architecture, multi-sensory elements, set works, extending the narrative before and after the visit, are suddenly part of the earliest discussions around the creation of the experience’s story.

As we grapple with what leading with story design will mean for our project, we’re facing further questions:

  • To what extent can the experience be set in an imaginary world but also have scientific accuracy?
  • Should robots be the primary focus or a “lens” for looking at other topics or themes?
  • Given that the experience will be unlike anything the museum sector has ever delivered before, how do we communicate what it is to audiences?
  • How can we place the visitor at the centre of the story to build a strong emotional connection with the narrative and let them help shape its outcome?
  • How might we extend the story before and after the visit to the museum?