XR for real - how Extended Reality is coming to life

Sandra: Jan, we’re talking about extended (XR) reality today. What is your personal background and experience with XR?

Jan: When I was studying about 16 years ago, I was already deeply interested in virtual reality, sparked by movies like The Lawnmower Man.

During my last two university projects, we experimented quite a bit with extended reality and developed a few things that became mainstream a few years later.

One of the projects was virtual ping-pong. We took a racket and glued markers on it. Using OpenGL, we then played ping-pong with it.

Four years later, Sony released the EyeToy camera for the PlayStation, giving it computer vision. I was sure that the technology would be everywhere within 3 or 4 years, but it took a lot longer, mainly due to technical constraints.

Fast forward to now, and we are at the stage where AR finally starts to gain traction. For adoption by the average Joe, hardware and software still have to improve, but the road to mass adoption is clear.

Sandra: Speaking of software and hardware development: who is driving the development of XR?

Jan: The biggest spender by far is Meta. They made investments to own the market. They might have been too early, but we’ll see. Sooner or later, it will come to mass adoption.

There are also newer players, for example, Apple. They also seem to invest quite a bit and plan to release devices soon.

Besides that, there are several smaller players, especially in PC VR. Many of them got crushed by the heavily subsidized headsets from Meta.

Sandra: What use cases do you see across industries?

Jan: Currently, the biggest use case is clearly gaming. This is where I see most of the development, but there are other interesting use cases besides that.

What really blew my mind is a new language-learning tool called Noun Town. It’s a VR app to learn Spanish, French, Italian, German, Japanese, and Chinese. New users start in a grey city and can pick up any object and hold it to their ear to hear its name. When they repeat it correctly, the object turns to color.

Soon after learning your first words, you can have short conversations with robots, like ordering food.

It’s superior to the old way of learning a language. Learning becomes a lot more fun and doesn't even feel like learning. Plus, it’s much, much quicker.

This is just one example of VR learning. VR has also been used for military and police training, and I see many other ways VR can help with education.

Another industry that uses AR is manufacturing, for example, for wiring complex machinery. Workers have AR headsets that show them where to mount certain parts.

In a different industry, Kinder also had a very nice idea: every toy in their Kinder Surprise eggs comes with a QR code to download Applaydu, a mobile app.

It uses the phone’s camera and augmented reality to bring the toy to life on your phone screen. You can see the toy running through the room, feed a toy animal, or watch several animals play together. My kids love this tech.

Sandra: I've seen the QR code inside the Kinder eggs before, and I'm gutted that I haven't scanned one before. It sounds as if it makes the toys much more enjoyable.

Jan: Definitely. Even I liked it quite a bit.

Sandra: Have you seen any new use cases that weren’t possible a few years ago?

Jan: It might be worth mentioning the massive advancements in mobile phone technology. Depth sensing has improved in the last 2 to 3 years.

For example, Apple’s high-end devices have a LiDAR sensor that senses the distance of objects across the room. The sensor uses a laser to collect measurements from your phone to the wall, to the chair, and so on.

Something else you can now do with these devices is taking a video of your environment and translate it into a 3D model. Our colleague Topher recently took a video of our office. I used the Quest headset to walk around in it, which was interesting.

Some of our clients have already started using this technology. A German insurer, for example, created virtual models of their office buildings. You can now visit every office in 3D and see what it looks like.

I think the real game-changer will be when you can have meetings in virtual models of physical places.

This is something we already see in gaming with the Meta Quest. Even if the overall virtual quality is relatively low, it is a different experience from having a Zoom call. It's evident what it means for the future: many meetings will be conducted in virtual reality.

I think that the whole aspect of virtual presence makes a big difference. The impact of VR on your brain is different from a simple 2D video. Soon, you'll start giving high 5s to colleagues you’ve never met before.

Sandra: While we were talking, you spoke about XR, AR, and VR. How are these terms different?

Jan: They are similar but different.

Virtual reality (VR) takes you into a synthetic environment using a VR headset. Everything is 3D-rendered.

Augmented reality (AR) overlays the real-world environment with digital objects, for example, using an AR headset or a phone's camera, like in the Kinder egg scenario.

Extended reality (XR) is the catch-all.

Sandra: You spoke about new headsets and technological advances like the laser sensor. How does this innovation happen, and what is one of the most memorable innovations for you?

Jan: Inventions come in waves. That's never a linear process.

Sometimes there are rapid advancements. And then there's a plateau when nothing happens. Then, all of a sudden, innovation spikes again.

We had a plateau phase for a long time, and the main reason was latency.

I had very strong VR sickness when I had my first VR headset 15 years ago. After 3 minutes, I was lying on the bed, wanting to vomit. A lag of 20 or 30 ms is enough to trick your brain into thinking you’ve been poisoned.

It took surprisingly long to have an interaction without any delay. On the other hand, it's a surprise that something like this is even possible. The technological advancements needed for real-time interactions are immense.

When you use the Meta Quest Pro, which has the best cameras, you see that the experience could be better, and there are still some breakthroughs needed on the hardware level, but with enough investment, they will come sooner or later.

Sandra: Speaking of Meta: are there any innovations related to the metaverse?

Jan: The metaverse is a virtual world that needs a lot of content. This content is rather difficult to create, but the whole process changes drastically as we speak. If you follow the news, you will have heard about all those breakthroughs with DALL·E  and Stable Diffusion.

While that’s only 2D, the jump from 2D and 3D is not far. And if you look at the research from Nvidia, you see that they are close to generating 3D objects, which will lower the cost of creating vast virtual worlds. Not 10x, but 1,000x.

Complex stuff like concept art, which took weeks to create, can now be created in a few seconds. Sure, it needs fine-tuning, but rather than taking weeks from start to finish, it now takes half a day.

Also, in the realm of AI, I’d like to mention the release of ChatGPT a few weeks ago. I guess it's an epochal shift for the workforce. ChatGPT is a text AI that you can ask any question. It can give you extremely detailed answers.

I think that companies like Stackoverflow will encounter difficulties mid-term because the answers from ChatGPT are often more precise.

This, in combination with image generation, will lead to a never-ending stream of content.

Sandra: Hmm. You already alluded to some things that might happen in the future. If you look into your crystal ball, what else do you see on the XR horizon?

Jan: With all the new AI tech, it will be very cheap to generate content, and I think that AI will generate everything at some point.

There are already the first prototypes to create movies using textual description, and eventually, that will translate into 3D worlds that you can consume in VR.

I think shopping is going to change as well. I see a massive opportunity for virtual fashion stores where you can try on clothes in AR instead of going to a store physically.

Then there are the super-futuristic or dystopian scenarios like brain chips. If you look at NeuraLink by Elon Musk, it seems as if they already have the prototypes that are more or less pending approval from the FDA.

Sandra: Okay, that sounds interesting. When you were talking earlier, I was looking forward to something like the holodeck from Star Trek, which sounds like something that might be possible.

Jan: The holodeck experience will be there. The Commodore 64 version is already here. Soon we will have the Amiga version, and 2 or 3 years later, we will have the full-blown iPhone experience.

This will also have quite a few social implications. I can imagine that, for some people, it will be more appealing to live their life in a virtual world. They’re probably going to disappear in their room forever.

Sandra: There is a fascinating film with Bruce Willis called Surrogates. Almost everyone has a remote-controlled android. These robots are better and younger-looking than their human owners. In the movie, humans hardly ever leave the house and society even looks down on people who do because their human bodies are less attractive and less fit than the androids.

When I watched the film, I thought that this scenario could become our reality at some point.

Jan: I think it’s comparable to sugar or nicotine addictions, and some people will find the virtual world so gratifying that they don’t want to live any other way.

Sandra: I guess it’s a question of perspective. I can see why this might seem attractive but I probably find it more frightening. I cannot imagine that living in a virtual world would make people happy unless we find a way to keep our bodies fit.

We can’t end here, so I suggest you share what XR use case you look forward to personally.

Jan: Having real meetings and getting rid of video conferences. That's something that I absolutely look forward to. The feeling that a person sits right in front of you is different than just looking at your screen.

And maybe virtual travel to check out places and meet people there. That, to me, is something that would improve many people’s lives.

Sandra: That would be nice indeed, and it also sounds very sustainable.

Sandra: Last question: thinking of the technologies that enable any of these use cases, what does this mean for Magnolia?

Jan: As we speak, we’re seeing a historical shift in digital content that will disrupt many industries and jobs. Text and images will be heavily automated, and we are very well prepared as a company.

We have first implementations that use ChatGPT for content writing and DALL·E for imagery. Our customers have confirmed that these solutions will reduce their efforts to create copy and images by 10x.

Regarding VR specifically, we enable new shopping use cases, where you can upload your product images in 3D, and customers can look at items from all angles and in their environment. This is already possible today.

We also have a simple use case where we place content in a virtual maze. You can walk through the maze and look at pictures and copy.

I give mass adoption of XR in the enterprise, for example, in ecommerce, another 1 or 2 years. We’ve done all the groundwork, so we’ll be ready to provide all the features you need to succeed with XR.

Sandra: Thank you, Jan. You gave us a good picture of where we are today, and what the future might bring.