Since the first iPhone, standard screens have been the medium for which we’ve been designing for most - usually apps and websites. As technology has advanced though, new avenues have slowly emerged (e.g. Augmented Reality and Virtual Reality) through which we can consume media and information in different ways.

"VR and AR are just 2 realms we can design for today – there’s actually lots more."

Many of these mediums involve blending the real world with digital information in quite unique ways. A popular AR example for instance, is IKEA’s upcoming Place app, which lets us overlay our living rooms with items of furniture to see how they’d look (or even if they fit in our house).

Augmented Reality (AR) and Virtual Reality (VR) are just 2 of the realms we can design for today though - there’s actually lots more, with each bringing a host of opportunities for app developers. Each realm is made up of a different combination of Time, Space and Matter - check them out in the contents below:

Contents: Realms of opportunity

1. Reality: Time, Space and Matter
2. Augmented Reality: Time, Space, No-Matter
3. Augmented Virtuality: No-Time, No-Space, Matter
4. Alternate Reality: No-Time, Space, No-Matter
5. Physical Virtuality: Time, No-Space, Matter
6. Mirrored Virtuality: Time, No-Space, No-Matter
7. Virtual Reality: No-Time, No-Space, No-Matter

A lot of the time, all these realms tend to fall under the umbrella phrase, 'Mixed Reality', but each term actually has its own meaning depending on the amount of reality and ‘virtual reality’ involved.

"Each different combination of Time, Space and Matter brings different opportunities."

In this article we’ll explore each one, and uncover ideas and inspiration for creating apps and experiences that combine Time, Space and Matter in different ways. It might give you some new avenues to explore when coming up with your new product! Let’s start by understanding this multiverse:

8 realms of opportunity

Reality and Virtual Reality sit at complete opposite ends of the multiverse, with a range of others realms fitting in between. They can be visualized on this octant, from the book, ‘Infinite Reality: Creating Customer Value on the Digital Frontier’ by B. Joseph Pine and Kim C. Korn, which this article is inspired by:

Eight realms of the Universe, by B. Joseph Pine and Kim C. Korn

Reality to Virtual Reality

The main difference between all of these realms of experience is the relationship between Time, Space and Matter. It’s more straight forward than it looks - let’s start by understanding the 2 extremes, Reality and Virtual Reality:

Reality is the realm against which we can benchmark and compare everything with. In reality, all 3 components are present: Time, Space and Matter. It’s the world we live in where we can move through space and touch physical objects.

"Reality is the realm against which we can benchmark and compare everything with."

The complete opposite realm is Virtual Reality, where Time, Space and Matter as we know in reality don’t exist. All 3 of these components are created depending on the virtual world being explored. It’s a bit like the Matrix - none of it is actually real.

In between these 2 extremes lie many realms, often simply summed up as Mixed Reality. However, each different combination of Time, Space and Matter brings different opportunities. For example, removing ‘Matter’ from Space and Time results in one of the most popular experiences today - Augmented Reality, where digital objects overlay the physical world. Let’s look into it:

Augmented Reality

We’re most familiar with AR these days. It’s when we have Time, Space, but the ‘matter’ doesn’t exist. We use the time and space from reality but overlay it with something digital, which presents us with a host of opportunities as outlined below:

Augmented Reality Opportunities

We’ve seen loads of AR over the last couple of years, with games like Pokemon Go being the most popular. Therefore, here’s just a couple examples since we’re already so familiar with these:

Online Retail

A popular example has always been to see how furniture would look in your house. Both Shopify and IKEA have been experimenting to make this a real possibility:

• Shopify:
  Shopify's technology has been used to power an app for Magnolia Market, letting you visualize how household products might look around your house before buying them. The app is built using Apple’s ARKit and the Shopify Mobile SDK, letting us preview products in our homes and purchase them with Apple Pay:
  

  Magnolia Market AR app, powered by Shopify

• IKEA Place:
  IKEA’s Place app lets you place ‘true to scale’ 3D models of IKEA furniture in your house. It uses Apple’s AR Kit to measure space and place furniture on surfaces. Once you’ve seen how the furniture looks, you can take advantage of existing features of the digital world by sharing images of the simulation with family and friends. Check out their YouTube video, it's awesome:
  
  "IKEA Place lets you confidently experience, experiment and share how good design transforms your space."

Tools for the World

As seen in both IKEA’s and Shopify's AR apps, being able to recognise space and surfaces with your phone’s camera makes AR experiences much more powerful. Using Apple’s AR Kit, we can:

• Access spacial measurements such as distance
• Recognise horizontal planes such as tables and floors
• Detect light in a scene, and apply it for lighting the virtual objects

These features make it possible to develop apps that can replace physical tools such as rulers:

TapMeasure, an AR App for measuring things

Gaming
As well as opportunities for tools and productivity, Augmented Reality provides a massive playground for games too. Unless you've been living under a rock, you'll have heard of a few so I won't go into detail. Think Pokémon Go for instance.

Where to start with AR

Overall, we’re augmenting what we can see in reality, with whatever technology we have available. At the moment, that’s often our phones, and therefore usually involves the camera of our mobile device. Therefore when it comes to building your own AR app, you can start with Apple’s AR Guidelines which has a host of tips for designing for AR using a phone's camera.

Another great starting point is this article by Designlab:

What Does the iPhone X’s AR Engine Mean for UX Designers?We also explore what AR could mean for UX designers, and, finally, our take on the $162bn question: are people going to use it?

Augmented Virtuality

Augmented Virtuality can be seen as an opposite to Augmented Reality in a way. Instead of overlaying the real world with digital stuff, we augment the virtual world with something from the real world. There’s less examples of this than Augmented Reality, but there’s still plenty of opportunities. One of the most prevalent examples of Augmented Virtuality are input devices like remotes and game controllers. A couple are the Nintendo Wii Remote, and more recently, Virtual Reality Gloves:

Nintendo Wii Remote

Augmented Virtuality Example: Nintendo Wii

In it’s time, the Nintendo Wii control was a gamechanger, bringing a new level of interactivity to games that wasn’t previously possible. Instead of pressing buttons to play a tennis game, we could actually carry out the motions of a tennis player and have it transfer to the digital realm.

Analog to digital

A key part that made this possible was the use of an accelerometer to detect direction and force applied to the Wii remote. That’s done by measuring movement along the X, Y and Z axis, as well as rotation. These analogue forces from the real world are then translated into digital data and sent to the Wii via bluetooth.

"Analogue forces from the real world are translated into digital data..."

Sensors that enable Augmented Virtuality

Watch this video to find out exactly how it works:

Virtual Reality Gloves

Moving on from the Nintendo Wii, and into a Virtual Reality setting, the same type of concept is present for VR gloves. However, the gloves not only detect the orientation of each hand, but also the bending of each finger using ‘flex sensors’. The number of sensors required increases with the amount of movement needed to be detected, which makes them much more complex. All the data collected from the gloves provide enough information to build a skeletal model of the hand.

"Data can be transferred from the digital world to the gloves, so users can feel items via haptic feedback."

VR Glove from Manu

Going further than that though, data can also be transferred from the digital world back to the gloves, so users can feel, touch and hold items via haptic feedback.

Manus VR have designed their gloves to be used specifically for virtual reality, but they could actually be used for any digital environment. Check out the video below to see the hand tracking in action through a Surgeon Simulator game:

There’s more information here on how the sensors work:

Getting started with Augmented Virtuality

To get started with Augmented Virtuality, you don’t really need any complex hardware. For example, here’s a guide to building your own Wii controller in just 10 minutes. And here’s a couple more projects from students, using every day objects from the physical reality to control digital worlds:

• Augmented Virtuality Game
• Augmented Virtuality Pong

Alternate Reality

Alternate Reality only interacts with the spaces of reality (e.g. fields, roads, surfaces). It’s quite similar to Augmented Reality, but in addition to a digital overlay on our world, time becomes fictional. It’s often based upon a storyline with it’s own concept of time. It’s defined pretty well in Wikipedia too, like so:

"An alternate reality game (ARG) is an interactive narrative that uses the real world as a platform, often involving multiple media and game elements, to tell a story that may be affected by participants' ideas or actions."

Essentially, events that normally take place online are brought to the real world across multiple channels. Popular examples from the past have often been marketing campaigns for product launches. Two notable ones are below:

1. The Dark Knight promotion for the Batman movie
   

   The Batman

   The Dark Knight ARG immersed over 10 million people in a ‘real world Gothem City’, where the story was played out through an array of different channels. From playing casual games, to receiving text messages and phone calls, players could unlock pieces of the larger story, experiencing an alternate reality. Check out this case study video that sums up the entire thing:
   

2. Another similar example is the I Love Bees promotion for Microsoft’s Halo 2 - Read about the story and game here.

Difference between ARG and AR

Since Alternate Reality Games are played out in real-world environments, they can be confused with Augmented reality games today, such as Pokémon Go or The Walking Dead. Pamela Hogle writes about the differences between the two in her article for Learning Solutions Magazine. A summary of those differences are outlined below:

Alternate Reality

• The game players have more control
• The story evolves depending on the actions and responses of players.
• Elements of the game are created to fit in with the real world, thereby reinforcing the real world (E.g. using phone calls and text messages for game clues).
• Without a story or narrative, the alternate reality games can’t exist.

Augmented Reality

• Players have less control - the entire content and story is determined by the game creator
• The game changes the view a person has of the world, augmenting it with something digital (e.g. Pokémon characters overlaying the real world).
• The game can exist without a story

Quick starting tip for Alternate Reality

As summarised in the table above, the key components for alternate reality games are a good storyline, and a network of users to play with. A good starting point could be this inspirational list of popular augmented reality games and software.

Physical Virtuality

Physical Virtuality is the combination of real time and real matter, but the space used is that from the digital realm (no space). It’s actually really common, enabling us to take advantage of digital environments to produce things for the real world. Notable examples are shops like Vistaprint, TShirt Studio, and even Moo. They all enable us to try out different designs and see exactly how they’d look before creating and purchasing the real thing:

"Items can be customised, and only actually made when a customer has paid for the design."

Physical Virtuality Advantages

Combining time and matter with no-space through physical virtuality brings lots of advantages that are useful in many contexts, especially within e-commerce:

• Reduce waste by prototyping: software allowing us to create digital versions of products reduces waste as we don’t need to use costly materials to get the design right.
• Reduce distance: 3D printing is a great example of reducing physical distances. Instead of ordering a product to be delivered physically, a design file can be sent over the web, from which the recipient can open and print the design directly in their own home on a 3D printer.
• Make on order: Items can be customised, and only actually made when a customer has paid for the design. This means an inventory of a specific item isn’t needed - materials only need to be used upon order completion.

Mirrored Virtuality

Mirrored Virtuality comes about when we map real world scenarios (that are happening in real-time) to the virtual world of no space and no matter. The best visualizations give us a compelling overview of information, making fast-paced information easy to consume. A great example is the EarthWindMap site, letting us see weather around the world:

Weather in the UK, from EarthWindMap

"The best visualizations give us a compelling overview, making fast-paced information easy to consume."

A simpler example is the ‘People living on Earth’ counter:

People living on Earth counter

It provides a simple counter of people on earth, which is quite strange to actually see in this way, yet easy to consume.

Essentially though, mirrored virtuality is offering a mirrored perspective of what’s happening in the real world. To get started creating one of these, you just need to find a data source - available from a host of APIs these days, and think of a way to visualize it. Check out this article for more inspiration.

Overall, Get Experimenting

Overall, we’ve covered a range of opportunities available to design across 7 different realms, and it’s clear to see that there’s plenty of scope for creating something totally new for your users to experience.

"There’s really no restrictions – combining different realms is also a possibility for new opportunities."

There’s really no restrictions - even combining the different realms is a possibility. For example, the VR Gloves by Manus combine Virtual Reality with Augmented Virtuality to create an even more immersive VR experience.

When thinking about this multiverse, it might also be worth to step right back and begin with the activity or experience you want to create, and then think about different tools that can help get you there. Maybe something as simpler as a smaller screen is enough to create a more immersive experience. As we saw with smartwatches, putting screens on our wrists changed the experience of consuming information quite drastically:

"On the Wrist..." by Luke Wroblewski

As shown in the diagram by Luke Wroblewski, smartwatches provided a new context to design for, allowing us to get updates and track health etc, whilst staying more connected with our surroundings. The experience was no longer focused around the screen, but also our activities and the world around us. The form factor of the watch leads us to only glance at its screen rather than getting totally lost in it like you would a mobile.

"When it comes to experimenting with VR environments, prototyping tools are a great place to start."

When it comes to experimenting with VR environments though, prototyping tools can be a great way to explore. For instance, it’s even possible to prototype VR experiences directly in Marvel - check out this article by Maxime on prototyping for VR with Marvel, and get started today!