Placing mixed reality in the museum

One of the main challenges of this project is to create two experiences taking place in two augmented worlds located within two museums that have very distinct spatial characteristics. Each experience is to be formed by a temporary installation comprising a set of rooms with specific architectural and scenographic qualities. This means that while the setting of each experience will feel a bit like an augmented set design, or possibly even an escape room, or a complex museum installation, it will, at the same time, form part of the broader visiting experience of each of the two world-leading museums.

To research how to build these spaces, and to understand what place they would occupy within the two museums, the audience research teams in the museums carefully considered exactly how visitors will be arriving at the two spaces, and what they will be encountering before and after each of the experiences. Museum visiting is, of course, part of a spatial, architectural experience, in which, as shown by architect Bernard Tschumi, ‘the very heterogeneity of the definition of – space, action, and movement – makes it into an event, that place of shock, or that place of the invention of ourselves’ (Ibid.: 258, added emphasis). The very architecture of the museum too, then, should be a place of action, of self discovery, and social cohesion.

Project team in Magic Leap headsets undertaking walkthrough of proposed experience

Each of the spaces we aim to build could be described as a world within a world that is transformed, on various occasions, into yet another world. Each of the spaces will in fact be located within one of the museums, and form part of their complex architecture, yet also be augmented through the use of the mixed reality.

To understand how visitors would move within such a complex space, and so have a better sense of the conditions under which the event would be produced, the team conducted a number of walk-throughs, testing how participants, who were team members on this occasion, would behave in narrow spaces when wearing a headset; how they might relate to each other; how easily they would understand and respond to instructions; what the timing of the experience might be; and how they would cope with being asked to take on different roles and acts as onlookers, participants, gamers, learners, teams, or even researchers at different points in time within the same experience.

Project team in Magic Leap headsets undertaking walkthrough of proposed experience

As each experience is structured around a specific use of space and time, it might be interesting to remember that the two terms are in fact strongly inter-related and constitute the fundamental concepts through which we define orientation. Human actions occur in time and through space, and, according to the Oxford English Dictionary, space denotes time or duration, and time is in fact a space or extent of time.

Each of the experiences entails the construction of a time present (the fictional here and now in which the visitor is placed), a chronological time (the narrative order of the events), and plot time (the inter-relation between the time present and the chronological time of the fiction), as well as a performance time (marking the duration of the overall experience). Spatially too the experiences will be located within the museums, and, within that, in a number of rooms, which will on occasion appear augmented.

Project team in Magic Leap headsets undertaking walkthrough of proposed experience

The complex hybrid space created by the overlap of these physical, fictional and digital spatio-temporal narrative structures are best described by using Katherine Hayles notion of ‘enfolded spaces’ in which what is witnessed ‘is no longer a homogeneous context for a given spatial area, but rather pockets of different contexts in it’ (in De Souza e Silva 2006: 269). The idea of a folding space cannot but bring back Gilles Deleuze’s theorisation of the fold, as an in-betweenness of spaces, able to represent dialectical opposites such as organic and inorganic, inside and outside (1993: 13). This folding spatio-temporal environment, which can no longer easily be represented in a two-dimensional drawing, or as a chronology, or even a conventional architecture, as it is only live as a mixed reality, constitutes a new kind of place within the museum.

References

  • De Souza e Silva, A. (2006) ‘From cyber to hybrid: mobile technologies as interfaces of hybrid spaces’, Space and Culture, 9:3, 261-278.
  • Deleuze, G. (1993) The Fold, tr. B. Conley, London and New York: Continuum.
  • Tschumi, B. (1996) Architecture and Disjunction, Cambridge, Mass.: MIT Press.

Prototyping at Factory 42

For the Factory 42 development team, the greatest feature of using Magic Leap – the element that pushes it ahead of any other augmented or mixed reality technology to date – is its ability to combine the physical and digital worlds together. The simple ability for digital content to react to physical objects in the real space: a table, the floor, even the player’s own hands, is a revolution in realism and interactivity. This concept is continuously at the heart of our designs and the prototypes we are developing.

For a few months now, the development team have been exploring the various functions and features of the Magic Leap One devices. Since then, we’ve put together prototypes and demos that test the breadth of Magic Leap’s features, including image tracking, hand and gesture recognition, eye tracking, telemetry and world reconstruction to name a few. We’ve even built our own online multiplayer scenes in which players can all wear Magic Leap headsets and share the same experience together.

Presented in this post are a handful of some of the features that are available with Magic Leap.

World Reconstruction

In order for our digital content to interact with the world around us, the Magic Leap device needs to scan its surroundings and build up a mesh of the environment; this is called World Reconstruction.

Once Magic Leap has created a digital version of the world we can experience true mixed reality.

Magic Leap One user's view of scanning of environment with mesh overlaid on office

Magic Leap Create app scanning environment in app setup
Create by Magic Leap

Image Tracking

The ability to recognise a specific image, and follow its position around the room, is a feature common to many mobile AR apps. Image tracking can be a useful way to activate your digital content at key locations in your scene; they can make pictures come to life.

View through Magic Leap headset showing tracking of an image on a desk

Important design notes to consider when using Image Tracking:

  • Image Tracking typically works best if the image is stationary and of a decent size; we don’t often go smaller than A5 if we can help it.
  • When deciding on which pictures to use, bear in mind that the Magic Leap image tracker discards all colour information, so try to choose images with high contrast.
  • Glossy or reflective finishes will interfere with detection of your image.
  • Tracking works best when the image is flat. Granted Magic Leap does not need to detect the entire image for recognition to occur, but bending or distorting your image around a 3D surface will increase the likelihood of errors.

Hand Recognition and Gestures

Hand Tracking, similar to Image Tracking, is the recognition of key points on the user’s hands. In addition to tracking all five fingers on both hands, Magic Leap is able to detect a number of specific hand gestures made by the player. These include: Open Hand, Closed Fist, Thumbs Up, Pointing, L shape, C shape, Pinching and OK.

Drawings of the eight hand gestures tracked by the Magic Leap One headset
Source: Magic Leap Creators Portal (June 2019)

View through Magic Leap One headset showing handtracking overlaid on user's hand

 

Accompanying visitors at the Natural History Museum

To assess what future audiences might enjoy as part of museum visiting, this project started by looking into existing studies exploring how current audiences experience museums and heritage organisations.

Two initial workshops were held at the Natural History Museum and the Science Museum, during which teams from the museums illustrated how they conduct audience research and researchers from the University of Exeter presented on business modelling based on the visitor experience, and the use of ethnomethodological-oriented methods in other projects (e.g. the EU-funded VISTA AR project and the AHRC-funded Performance at Tate project).

It is common practice in museums, galleries, heritage organisations, and in many other experience-based organisations, to use ethnographic methods to study audience behaviour.

Visitors and exhibits in Hintze Hall, Natural History Museum, London, including a blue whale skeleton is suspended from the ceiling.

Ethnographic methods are also widely used in the study of mixed reality. For example, the Mixed Reality Lab at the University of Nottingham has devised a series of ethnomethodologically-informed methods, including a technomethodology [PDF], which are aimed at assisting researchers at the Lab to understand the challenges in the design of specific human computer interactions.

It therefore seemed that the project could start by conducting an assessment of the current visitor experience at the Natural History Museum by using accompanied visiting.

Accompanied visits use some ethnographic strategies to observe what visitors do in museum, galleries or other heritage organisations. During an accompanied visit, a member of staff accompanies one or more visitors, either observing them or engaging more directly with them, while annotating spontaneous behaviour and conversations. It is known that accompanied visits are a great way to give voice to visitors and are particularly useful when studying groups from under-represented backgrounds (Haywood 2018).

On this occasion the museum audience research team had asked project members to look into what the museum experience was like for families and adults under 35, what people did, whether there were moments of confusion or obstacles and barriers, and to reflect about what conclusions and lessons could be learnt from this experience that could be applied to the project more broadly.

Visitor at Natural History Museum, London, viewing text and information panel on human sex cells.

I accompanied a couple, a man and a woman, in their late twenties. They had both visited the museum as children but had not returned for a number of years.

They first headed to the human biology exhibition where I first noted that they did not take photos, though they did buy a souvenir.

I did not have permission to photograph the two visitors, but had permission to capture generic visitor behaviour, and found this was usually consistent with the behaviour of the two visitors I accompanied.

Visitors at Natural History Museum, London, including information panels and two dinosaur skull exhibits.

It was noticeable that visitors often struggled to get close to the exhibits due to the large number of people in each room, which often resulted in moving more quickly than anticipated to another exhibit, or even a different room.

Visitors at Natural History Museum, London, viewing marine vertebrae exhibits.

One of the visitors I accompanied was interested in seeing the marine vertebrate gallery and looking at the dinosaurs, and so both visitors spent a long time in the marine vertebrae exhibition, reading the interpretation carefully, often discussing specific artefacts with each other.

Visitors at Natural History Museum, London, viewing and photographing T.rex exhibit.

Once they reached the roaring T. rex animation, they stopped, observing that the exhibit offers a photo moment, but then they walked quickly through the remaining dinosaur galleries as they had to leave to for a meeting.

Visitors at Natural History Museum, London, viewing exhibits.

Other groups, accompanied by other project members, showed some interest in sensory experiences, and younger people especially liked touching exhibits.

As Dani Parr found by following her group, audiences ‘expect to learn every time they come to the museum’. In fact, for all groups, there was an expectation to learn, to see something new, and a great excitement about interactive and responsive exhibits.

Exhibits at the Natural History Museum, London, including blue whale and rhinoceros.

The only two ‘wows’ in my group occurred, respectively, in the Hintze Hall, when the whale was first spotted, and when I explained that the aim of the project was to experience a dinosaur in mixed reality.

Reference:

Naomi Haywood (2018) ‘Accompanied Visits as a Tool to Understand Visitors’ Experiences: A Critical Reflection and Proposed Typology’, Visitor Studies, 21:1, 135-147, DOI: 10.1080/10645578.2018.1503876

Creative workshops

Workshop whiteboard with the word audiences written on it
Understanding future audiences was one of the drivers of the creative workshops held at the Science Museum and the Natural History Museum.

Since our project started a few months ago, various teams, each representing an indispensable area of expertise, have been researching what audiences of the future might wish to engage with and how best to facilitate this engagement. Crucial questions have been asked including, for example, how to provide audiences with valuable learning experiences in mixed reality; how to iteratively test with audiences; how to capture what individuals and groups might experience in a mixed reality environment; and how to ensure the widest possible reach and accessibility of our work.

Bethan Ross, Senior Audience Researcher at the Science Museum Group introducing the team to audience profiling practices at the Science Museum.
Bethan Ross, Senior Audience Researcher at the Science Museum Group introducing the team to audience profiling practices at the Science Museum.

The project started with a series of creative workshops, which were held at the Natural History Museum and the Science Museum, and facilitated by the immersive and content production studio Factory 42.

Documentation of the first creative workshop at the Science Museum.

Documentation of the first creative workshop at the Science Museum.

The first two workshops were informative and served the purpose of introducing the various teams involved in the project to existing research. It allowed us to explore the multi-sensory possibilities of storytelling in Magic Leap; brainstorm possible wow factors; and identify possible challenges by conducting multiple risk assessments. The workshops were extensively documented to create a rich resource for those who might be interested in researching the work in years to come.

The Science Museum creative workshop lead, Dani Parr, overlooking the team’s exploration of what they thought a robot could look like.

These two initial workshops were followed by a further set of creative workshops in each of the museums. The first day was about imagining what worlds could be built. The second day consisted of a writing workshop and was led by an expert team of creative storytelling writers. The third day focused on working through some of the stories, paying particular attention to the technological challenges, and included a workshop with museum staff looking at a number of stories, and the fourth day was about storyboarding the three strongest options, focusing on branching narratives and audience agency. Finally, during the last day two stories were selected so that they could be shared and discussed with museum staff and the wider team. The feedback indicated at this point that both stories should be kept ‘alive’ to test Magic Leap opportunities before making the final selection.

Documentation of the brainstorming workshop about robots at the Science Museum.

Documentation of the brainstorming workshop about robots at the Science Museum.

Both creative workshops were designed and led by the experienced theatre director and participation producer Dani Parr from the Almeida, who involved participants in thinking about which kinds of robots or dinosaurs we might expect to see in Magic Leap; how groups would work together in mixed reality; how the chosen space within the museum would work in practice; what active learning could mean in this context; how the experience could be accessible and give a sense of exploration while also being semi-scaffolded and orchestrated on the day; how the story might be extended before and after the museum visit; and what would be the relationship between fact and fiction, imagination and museum learning.

Discussions focused on how the design of the experience would need to appeal to people from different age groups, different cultures, different competences; work for families, friends, and individuals; have multi-sensory triggers; be accessible and scalable for touring; encourage repeat visits; have a wow factor and be scientifically accurate.

What is mixed reality?

The term mixed reality indicates an overlap between a physical and a virtual world produced through the use of a range of more or less immersive technologies.

Diagram showing scope of mixed reality (MR) with four points on a line running horizontally. Four points are marked on the scale. Left to right these read: 1 Real Environment, 2 Augmented Reality (VR), 3 Augmented Virtuality (AV), 4 Virtual Environment. Above this scale Mixed Reality (MR) is show and encompassing a wide range of the scale from 1 to 4.

The expression was first discussed by Paul Milgram and Furnio Kishino, who proposed the idea of a ‘mixed reality continuum’, which connects what they called ‘real’ and ‘virtual’ environments, including intermediate points spanning from ‘augmented reality’ to ‘augmented virtuality’ (1994).

Mixed reality has been successfully used in a number of contexts spanning from training to medicine. Within the creative industries and the heritage sector, mixed reality has been researched extensively by Steve Benford and his colleagues at the Mixed Reality Lab at the University of Nottingham, often in collaboration with the Brighton-based performance group Blast Theory. Their pioneering use of a wide range of interfaces has been published extensively in HCI journals, and the framework derived from a decade of practice-research has been analysed in Performing Mixed Reality (Benford and Giannachi 2011).

Blast Theory member Ju Row Farr emerging from behind the rain screen, Desert Rain (1998)
Blast Theory member Ju Row Farr emerging from behind the rain screen, Desert Rain (1998), Copyright Blast Theory.

The design of the mixed reality experiences is particularly complex. This is because, as the Lab’s research has shown, mixed reality environments often occupy various points on the continuum, which could be adjacent to each other so that users could, for example, look or even pass from one to the other, as was the case in Blast Theory’s Desert Rain where the performer, in a coup de théâtre, walked through the projection of their image to hand a magnetic swipe card to the participant (1998).

Science Museum Managing Director Jonathan Newby testing Magic Leap.
Science Museum Managing Director Jonathan Newby testing Magic Leap.

Mixed reality is usually experienced through see-through head-mounted displays, whereas augmented reality tends to be experienced by pointing a hand-held display at a physical object, which then augments the object. In each case, the computer system is able to track the user’s location and augment their sphere of vision digitally. The use of the Microsoft HoloLens and the recent release of Magic Leap, the technology used in this project, have produced a lot of interest in mixed reality, indicating that we will be likely to make increasing use of mixed reality in years to come.

While museums have often used virtual reality, the use of mixed reality is still relatively new, though recent examples developed by Microsoft include the experience of a Ford car at the Petersen Locomotive Museum (2018); encounters with old Japanese artworks at Kennin-Jim, the oldest Zen temple in Japan (2018); and an encounter with an astronaut in Defying Gravity exhibition at the Smithsonian (2018). Interestingly the latter allowed four people to be in the experience at the same time, making it possible for families and friends to visit together.

In this project, we will be using Magic Leap’s head-mounted virtual retina display, which superimposes 3D computer-generated images over real world objects. The display works by projecting a field of light into users’ eyes. When testing Magic Leap, one could  literally see the physical world and, blended within it, the digital world created through Magic Leap. Interestingly, the digital world does not look like a film, but rather like sets of 3D entities, making it possible for users to establish a strong sense of presence in both the physical and digital worlds.

This project aims to create two new environments which will be designed for the Science Museum and the Natural History Museum using Magic Leap. The design of exciting content is paramount so as to ensure the experiences will be engaging and memorable. A large team of experts comprising museum researchers and curators, academics from a range of disciplines, technologists, producers, participation experts, creative content providers, sensory storytelling and education experts, among others, has been brought together for a set of creative workshops which will be the topic of the next blog post.

Thoughts on mixed reality at the Science Museum

The Science Museum has a long, long history of interactive exhibits and use of digital technology. The earliest explorations of how hands-on experiences could bring science and technology to life date back to the 1930s.

Image of 1930s interactive gallery
Schoolboys in the Children’s Gallery of the Science Museum, March 1934

Interactivity is now a fundamental part of the museum’s offer and making sure that these exhibits are audience centred and fun is something we take incredibly seriously.

Interior image of Wonderlab: The Equinor Gallery at the Science Museum
Wonderlab: The Equinor Gallery at the Science Museum

The museum also adopted digital experiences early and remains a leader in this field. In recent years the museum has begun to explore the opportunities afforded by immersive technology to bring objects to life—to explain how they work, their historical context and the lives and experiences of people who used them. So, far we’ve explored Tim Peake’s return to Earth from the International Space Station in the Soyuz descent module in VR…

… and how VR can explore the complex mathematics behind the design of the Handley Page aircraft which is the centrepiece of Mathematics: The Winton Gallery.

Over the past ten months, driven by the Audience of the Future: Demonstrators application process, we’ve had dozens of conversations exploring the exciting affordances of mixed reality in the museum. When colleagues have asked us what we’re creating we’ve had to tell them that it is about robots and that the experience will combine: museum exhibit, digital environment, adventure game, immersive theatre, and escape room. So, here are our early thoughts and questions which are shaping our thinking:

1. Interactive experience

The two VR pieces that the museum has produced to date are linear. The experience unfolds and the audience member watches and listens; they are fundamentally passive. The opportunity for our new experience to offer agency to the audiences is incredibly exciting as we know that interactivity increases audience enjoyment and learning.

However, interactivity presents the project challenges too:

  • To what extent is the experience a sandbox environment that audiences can explore and to what extent is the experience on rails?
  • What does this mean for the experience design?
  • How do we inboard audiences to an interactive experience which will for almost all be their first use of mixed reality? Can this be part of the experience’s narrative?
  • How do we ensure the experience is inclusive and accessible?

2. Social experience

For most audiences, visiting the museum is a social experience. People come in groups. Whereas VR is an isolating, solitary experience, mixed reality has the potential to create a shared experience in which a group of visitors can collaborate or experience something together. This is hugely exciting for us as we can realise all the immersive benefits of VR but with this critical social element.

Again, there are challenges here too:

  • What if the group know each other well?
  • What if they don’t know each other?
  • What if they don’t speak the same language?
  • What if someone undertakes the experience alone?
  • Could the experience work for groups of school children?
  • If one or more member of the group can’t use the mixed reality headset for whatever reason, how can they still be part of the experience?
  • Are there opportunities for “Instagram moments” that can be shared on social media with friends and family beyond the walls of the museum?

3. Story-based experience

Stories are an effective way of connecting visitors with the museum’s displays. The way the museum traditionally approaches stories—called “narratives” internally—is centred around objects from the museum’s collection and the stories they tell, delivered through architecture, exhibit design, interpretation and programming. Our approach with this project will be completely different. Here we are beginning with the experience design and the story. There will be no physical objects—though there will probably be some 3D scans.

This approach really flips the museum’s traditional ways of thinking and working on their head. Conversations about experience design, architecture, multi-sensory elements, set works, extending the narrative before and after the visit, are suddenly part of the earliest discussions around the creation of the experience’s story.

As we grapple with what leading with story design will mean for our project, we’re facing further questions:

  • To what extent can the experience be set in an imaginary world but also have scientific accuracy?
  • Should robots be the primary focus or a “lens” for looking at other topics or themes?
  • Given that the experience will be unlike anything the museum sector has ever delivered before, how do we communicate what it is to audiences?
  • How can we place the visitor at the centre of the story to build a strong emotional connection with the narrative and let them help shape its outcome?
  • How might we extend the story before and after the visit to the museum?