With the announcement of Apple Vision Pro and the expansive XR experience at SXSW, excitement around extended reality is greater than ever.
The new Apple Vision Pro headset announced recently is among the biggest in tech news. While Cupertino has been quick to distance this hardware from VR and the metaverse, the power of this announcement has reignited interest in tech-enhanced immersive experiences.
The Apple Vision Pro is Apple’s first augmented reality (AR) headset. It has two 4K micro-OLED displays with a total of 23 million pixels, which is nearly three times as many as a 4K display. It also has a camera and sensor array and dual Apple silicon chips. The Vision Pro turns a physical room into a virtual space by mapping it with its front-facing sensors and cameras. You can use the headset as its own computer to browse the web and move around 4K windows with your hands.
Unfortunately, for most of us, the Vision Pro will be initially priced out of our price range, retailing for a whopping $3,499, which is close to the cost to make it. The three most expensive components are the camera and sensor array, the dual Apple silicon chips, and the twin 4K micro-OLED virtual reality displays.
Apple announced the Vision Pro at its June 5 WWDC keynote, but it won’t be available until next year.
I will admit that XR (extended reality) — which is a blanket term encompassing VR (virtual reality), AR (augmented reality), and MR (mixed reality) — has always been a fascinating topic for me. But that fascination, until recently, was only casual.
I tried some early VR games and basic AR experiences on the phone. And I thought the technology was certainly cool. But it was never anything I became fully immersed in. At the time, it was more like a fun but short-lived gimmick that was enticing for how new and different it was, but it didn’t seem to change the landscape of immersive and interactive media fundamentally.
All that changed when I attended this year’s SXSW XR Showcase, as well as several of the XR-related discussions and events.
Before I delve further into this topic, let me clarify by defining some of the key terms I’ll be using.
Most of us, especially gamers, are most familiar with VR. Augmented reality (AR) and virtual reality (VR) are both technologies that enhance the real world. AR uses a real-world setting, while VR is completely virtual. AR users can control their presence in the real world, while VR users are controlled by the system. AR is 25% virtual and 75% real, while VR is 75% virtual and 25% real. AR requires higher bandwidth than VR. AR can be accessed with just a smartphone, while VR requires a headset device.
Mixed reality (MR) is a term used to describe the merging of a real-world environment and a computer-generated one.
Physical and virtual objects may co-exist in mixed-reality environments and interact in real-time. It’s a hybrid combination of augmented reality and virtual reality technology. Mixed reality headsets like the Microsoft HoloLens are opening up possibilities in the gaming world for mixed-reality experiences. Mixed reality devices are different from typical VR hardware because they use mounted translucent glasses like the recently discontinued Google Glass that let us stay grounded in the real world.
XR is an abbreviation for Extended Reality. It’s an umbrella term that covers VR, AR, and MR. These technologies merge the physical and digital worlds, and they can enhance or replace our view of the world. XR can do this by overlaying or immersing computer text and graphics into real-world and virtual environments or even a combination of both.
You’ll start to hear this term being used much more frequently as technology advances and the world of potential uses and possibilities for this technology is greatly expanded.
Now, why am I telling you all this?
During SXSW 2023, I was invited to a press preview of the festival’s massive XR Experience. I spent several hours exploring some of the mind-blowing technology (and I could have spent several more), and I could not believe how far XR technology has advanced and how profound the implications of these advancements really are.
Highlights of the XR Experience at SXSW 2023
SXSW is considered one of the biggest festivals and conferences in the world. It takes over the city of Austin each spring.
The SXSW XR Experience is an exhibition at the 2023 SXSW Conference & Festivals that showcased immersive projects by international producers. The exhibition showcased virtual reality, augmented reality, and mixed reality technologies that promise to enhance our ability to perceive and experience the world around us.
Having the chance to experience first-hand some of the cutting-edge technologies and innovative use cases for this technology really opened my eyes to what’s possible. I am a gamer, and I’ve always been fascinated by the application of VR within the world of immersive gaming. Make no mistake, there was a ton of exciting content on display at SXSW that made the gamer in me positively giddy.
The capacity to thrill, terrify, and challenge players is now greater than ever. But what really wowed me was all the applications of this technology beyond just gaming.
TEMPORAL WORLD
With “Temporal World”, Berlin-based artists Chloé Lee and Lucas Martinic have created a haptic VR experience in which they reflect on the meaning of places with which they have varying degrees of connection — from the personal to the familiar to the foreign. As an Asian-American artist, she explores themes of migration through discovery, documentation, and, eventually, rooting herself in a new place.
We are guided to slow down while exploring Berlin and beyond in our increasingly accelerated time. This VR world emphasizes the ephemerality of memory. Visitors explore and shape the landscape that is as fragmented and fickle as memory itself while wearing a custom haptic jacket that adds sensation to the typical audiovisual VR experience. These new ways of sensing and, in turn, understanding help create a more nuanced understanding of our interactions and, often invisible, dynamics within our environments.
As Lee explains:
“Every day, though we may not see it, our smallest interactions shape and leave a trace on the world around us. This virtual world is not unlike our physical world; with distance and time, we often gain clarity. This project is an embracing of temporality. Although the places, people, and things that hold the essence of meaning are impermanent, the trace left behind can always be reimagined into something new.”
While wearing a custom haptic jacket that sends vibrational feedback through the body, participants enter the Oculus headset to find a single-user virtual world composed of 3D-scanned sights and soundscapes of Berlin and beyond. This jacket is an attempt to recreate a vibrational experience in which the highest frequencies are felt in the neck and shoulders, while the lowest is felt in the fingertips and waist area.
AESPA VR CONCERT AT KWANGYA
I’m a huge K-Pop fan, and I love the girl band aespa from South Korea. Thus, I was especially excited to check out a clip from their upcoming immersive VR concert, and it was, to quote the title of one of their most popular songs, Next Level.
The concert, which made me feel like I was front row for a live performance, was produced by AmazeVR, which brings fans face-to-face with their favorite artists. Using live-action 3D footage combined with beautifully rendered virtual environments, AmazeVR Concerts transport fans and artists alike to fantastical new worlds. AmazeVR recently launched the “Megan Thee Stallion VR Concert” experience and will be launching “AmazeVR Concerts” across multiple VR platforms this summer with multiple artists. I seriously cannot wait until this becomes the norm and would have loved to have had way more than just a few minutes of experiencing this remarkable technology.
SM Entertainment & AmazeVR’s first joint production utilizes new proprietary technology to produce immersive VR Concerts for artists globally at scale. The World Premiere of aespa’s VR Concert at KWANGYA was showcased in Austin, Texas, at the 2023 SXSW XR Experience March 11th – 14th, 2023. This experience will allow aespa fans to engage up close and intimately with both the artists and their stories in a way not possible anywhere else.
Forager: Immersive Multi-Sensory Experience
Perhaps no experience affected me more than this one, which was designed to be educational in nature but also made me feel more at peace and meditative than I’ve ever felt. In this experience, the developers take you through the entire life cycle of mushrooms in VR. Through sight, touch, and smell, you experience the progression from spores, mycelium, and fruiting body to decay.
A mushroom is a door in the floor. Open it and discover nature’s greatest revelation: an endless underground network secretly feeding life to the world above. And yet, we know more about the surface of the moon than we do about mushrooms.
What if we knew more? Enter Forager, a highly immersive experience and first-of-its-kind interactive volumetric time-lapse installation. Participants experience the wonder of the mushroom kingdom through a variety of different perspectives, ultimately fostering a closer connection with the natural world. Forager was created by Winslow Porter and Elie Zananiri, two well-renowned storytellers and technologists who have been breaking boundaries in virtual reality and installation design throughout their practice.
They have been developing Forager over the past two years while working as artists-in-residence at Smallhold, the only organic mushroom farm in New York City. The team’s goal is to use advanced storytelling tools to convey an urgent message: we are deeply connected to our planet in a way that has never been more important to understand than now.
The Forager team developed custom 3D scanning hardware and software to capture mushrooms growing over time and then play them back as hyper-realistic animations. This is the first ever volumetric time-lapse technology used inside of a game engine. This unique pipeline allows Forager to present the natural world as close to reality as possible, with the goal of sharing this connection to nature with all participants of any age, language, and background.
Rockets, by Pillow
And now, I bring you the game that made me immediately want to go out and buy a VR headset.
“Rockets, by Pillow” is a 2023 VR narrative designed for lying down in bed. In the game, you control a rocket named Crimson through puzzles to save the universe from an evil you set free. The game was created by Lucas Rizzotto and Chris Smoak.
You control the experience with just one controller. It’s just challenging enough to keep you engaged without being overly difficult or frustrating. It’s also insanely relaxing and would be the perfect way to unwind at the end of the long day as you get ready for bed. It’s enough to stimulate the brain without being so stimulating as to make it difficult to sleep afterward. The graphics were phenomenal, and it was also very funny.
I honestly am obsessed with this game and wish I could purchase it immediately.
More Phenomenal XR experiences
- Body of Mine VR: VR experience that, in combination with real interviews of transgender people, lets you experience your own body from a different perspective.
- Consensus Gentium: Interactive film that explores facial recognition, artificial intelligence, and the effects of uncontrolled surveillance.
- The District VR: German VR experience that immerses you in a digital twin of Berlin’s club scene. You become a DJ and make the virtual party-goers dance to your own beats.
- El Beat: A VR experience that lets you experience a journey of enslaved African men during the colonial era from the coast of Senegal to the Palenque area.
- Find WiiLii – Ep.1 The Gate-Crasher: In an alternate reality, humanity uses a teleportation system where there is a five percent chance that travelers will lose their memory. As part of a service team, you set out to find lost memories.
- Fresh Memories: The Look: This VR short film shows you the extent of the war in Ukraine through the eyes of those affected. You’ll be guided through the bombed homes, schools, hospitals, and streets of Kharkiv.
- The Invited: reimagining the Dracula story in augmented reality. An animated pop-up book shows you Dracula’s entrance into the modern world, which fills with new AR holograms with each turn of the page.
- Jailbirds – The Eye of the Artist: The interactive story concludes the Jailbird trilogy about the peaceful giant Felix, ending the story where two different concepts of justice collide.
- JFK Momento: An immersive documentary about the assassination of U.S. President Kennedy. You’ll experience the investigation surrounding the assassination, see original archival photos and films remastered in 3D, and hear interviews from the last living eyewitnesses.
- Once a Glacier: In the Inupiaq tradition, glaciers are carriers of memories from the past, which they convey in a song. This VR film shows you a girl’s life journey that parallels the end of the glacier’s ice, making the consequences of climate change tangible.
- Stay Alive, My Son (Chapters 1 & 2): You’ll experience the journey of Yathay, an old man who had to leave his son behind while fleeing the genocide in Cambodia. Along the way, you’ll delve into locked memories and accompany Yathay as he tries to reunite with his son.
- Whipped Cream “The Dark”: The Dark combines electro sounds and opera music to tell the emotional story of a toxic relationship.
Highlights of the XR sessions at SXSW 2023
In addition to the interactive XR experience showcase, there were a large number of XR-related discussion panels. You can catch recordings of the following inspiring events (links to the recording sessions below) that helped illuminate the future of XR, explaining where we’re heading and why it’s so important.
Augmented Realities: XR Innovation in Live Events: Live events — especially music festivals and sports — are a perfect showcase for the ultimate potential of this coming mixed reality (XR) spectrum. Industry experts from Epic Games, Coachella, and more discuss the many upcoming possibilities. (Audio only)
Highlights:
- Coachella has 400,000 onsite over the course of a weekend but another ten million watching on the live stream. How do you make these huge events feel like a unified fan experience and make them just as immersive for remote fans?
- AR is one of the biggest technology trends right now, and it’s only going to get bigger as AR-ready smartphones and other devices become more accessible around the world.
- Technology is being used to enhance existing broadcasts to make them feel more immersive and visually compelling, like the giant mixed-reality Panther used by the Carolina Panthers at their Home Opener that went viral or the 7-minute multi-cam MR production that aired live before the 2020 Superbowl kick-off.
- Technology is helping democratize the user experience, allowing those who aren’t able to attend these events in person due to geography or financial limitations to still experience the immersive magic of the live show from a virtual setting. Meanwhile, XR is helping take live events to another level, giving attendees the chance to experience a more multidimensional show-within-a-show.
Building a Sustainable Economy in the Metaverse: 2022 SXSW Hall of Fame inductee Nonny De La Peña leads an all-star cast of panelists covering what the metaverse is today, as well as where this medium needs to go to move beyond growth and create a utility that will help maintain a sustainable economy.
3D is no longer just for gaming. We are in the midst of a massive digital revolution from 2D to 3D. The metaverse is changing how we connect, work, and play. As the metaverse is being created, it is our responsibility to build a sustainable foundation. Join 3D, AR, and VR visionaries, including Nonny de la Peña, Dan Burgar, Ashley Crowder, and Cortney Harding, to discuss what the metaverse is today and how to take action to move beyond growth to create utility and maintain a sustainable economy. They will explore how to build an inclusive environment, drive a shared value for businesses and creators, empower consumers, and enable ways to measure impact to create digital sustainability. (AUDIO AND VIDEO)
Highlights:
- The metaverse is the next evolution of the web and how we communicate. We are going to start to see more and more 3D and AR everywhere on the web, including e-commerce. We will see more advanced headsets and technology, but the phone will continue to be our primary source to access this new technology.
- Within 3-5 years, this will be much more normalized. It won’t be considered new or different. It will be expected and well-integrated into our everyday life. Gen Z will lead the way.
- People will be able to make immersive content without knowing how to code.
- This technology will be instrumental in driving social impact and change.
Is This Really Live? Performance in the Age of XR: This session dives into the assumptions we make about “live” and explores how tools and techniques used in XR Theatre allow us to create sensations that mimic the in-person experience. (AUDIO ONLY)
XR theater is a type of theater that uses immersive technology, such as augmented reality (AR) and virtual reality (VR). XR theater has been used to create new kinds of live and virtual experiences. Check out this XR Theatre demo to get a better idea of the technology in use.
Highlights:
- We are able to blur the lines of technology between what is happening in real-time versus what is pre-recorded/pre-rendered.
- Work is being done to meaningfully engage XR into the narrative and not just a gimmick.
- Accessibility is key. People from all over the world, regardless of means, can experience these productions in some meaningful way.
- One of the projects discussed is ChekhovOS, an experimental game/performance experiment inspired by Anton Chekhov’s The Cherry Orchard and drawing from Chekhov’s letters and dreams. Learn more here. Another project discussed is the Black Queen VR Theater, and you can view more here.
- Even simple XR elements can dramatically enhance the user experience.
Metawashing!? A Futurists View on the Metaverse: The metaverse is the talk of the town, but how will it unfold? What might the different possible futures look like? How is this going to influence your life? And how will these developments affect the way we live? Get a unique insight into four future scenarios made by the Copenhagen Institute for Futures Studies, and learn what experts around the world find most plausible. (AUDIO ONLY)
Get a unique insight into four future scenarios made by the Copenhagen Insitute for Futures Studies, and see what experts around the world find most plausible from a brand new Delphi study on the development of the metaverse towards 2030. This is the first-of-its-kind global study seeking to source the collective knowledge and perspectives of a diverse panel of experts on the future of the metaverse. The Institute also released a Whitepaper on The Future of the Metaverse. More information about the Delphi report and the White Paper and links to download the documents can be found at https://cifs.dk/metaverse/.
Highlights:
- The metaverse is the seamless convergence of our physical and digital lives. We don’t yet understand the metaverse or what it’s capable of achieving, but the time to help shape the future of what it will become is now.
- The metaverse is much bigger than one company or one technology. It’s not yet here in any meaningful way, at least not the mature metaverse, but that doesn’t mean it won’t be a massive game-changer.
- We are going from being ON the internet to being IN the internet.
- Most experts believe the metaverse will have a massive future impact on society.
Reality Check: Are VR and AR Ready for Prime Time?: More VR/AR headsets are expected to ship than gaming consoles by the year 2025. In this session, immersive technology leaders contemplate how extended reality will revolutionize a variety of consumer experiences. (AUDIO ONLY)
For decades, VR and AR were the stuff of science fiction and either impractical or unaffordable. Today, these technologies have “crossed the chasm” and — thanks to tech giants like Meta, Apple, and others — are predestined to jump from niche experiences to go-to consumer platforms for the Metaverse.
Highlights:
- Apple’s entry into AR/VR could significantly change the landscape and legitimize the industry, taking XR into the mainstream. Iteration will drive more adoption.
- It’s very hard to get the technology right, but it’s even harder to create real and meaningful engagement.
- The trick is to create immersive experiences you can’t have in the real world, not just limit it to simple simulation.
- VR won’t die; it’s had about three hype cycles and keeps coming back to the forefront of exciting new tech.
Other Great Sessions From SXSW 2023 to Check Out
Simulating the Future: New Metaverse Forecasts: Metaverse technologies have the power to change many aspects of our lives and world, from how we govern to how we entertain ourselves to how we tackle the world’s greatest challenges. Simulations allow us to create and manipulate “virtual atoms,” paving the way for new paradigms, economies, and ways of being human. This presentation takes you to the edge of what’s emerging to redefine reality and the futures we might inhabit.
Synchronous Realities: Live XR Entertainment: Covering a variety of platforms and technology available to creators and designers, this session offers insights on how to ignite your audience with the next generation of immersive experiences.
The Ultimate Potential of VR: Promises & Perils: Over the last eight years, Kent Bye has asked thousands of virtual reality creators, “What is the ultimate potential of VR?” This session shares some of the best answers that map out the future of spatial computing, immersive storytelling, and experiential design.
Watercooler to Headset? The Future of Workplaces: Not only has XR-powered workspaces transformed enterprise management, but it has also gone the extra mile in removing hurdles for persons with disabilities. Inclusive design in immersive technology has created a universal experience that integrates tools that all users can enjoy. Join panelists for a live demo of XR technology and a discussion on the road ahead for the future of work.
XR Roundtable: The Future of Eyewear in Practice: This session brings together innovators at Verizon, Niantic, and Magic Leap to explore both present and future trends in HMD applications. Spanning both consumer and enterprise solutions, from personal computing to remote work, to multiuser gaming, hear how professionals are thinking about, preparing for, and executing the upcoming era of HMD-based augmented reality.
Follow Us!