Exploring Meta’s Quest Integration for Unity: My Experience
My first steps started by experimenting with Meta’s Quest Integration for Unity. Here’s a quick rundown of my journey and the challenges I faced.
Easy Setup for Quick Prototyping
The initial setup was straightforward and user-friendly. Meta provides a lot of built-in functionality, making it ideal for quick prototyping. Getting a basic project up and running took minimal effort, thanks to the clear setup guides.
Complications Arise with Advanced Features
However, as I moved beyond the basics, things became complicated. The documentation from Meta is often outdated or incomplete, making it difficult to implement more advanced features.
What I Built
I created a terrain with high-definition textures and added functionalities for teleporting (locomotion) and interacting with objects. The process involved using Unity’s XR Interaction Toolkit, which, despite some confusing moments due to lacking documentation, helped achieve the desired interactions.
Performance Issues
A significant issue I encountered was performance. Despite the simplicity of my scene, I noticed major frame rate drops, falling below 38 fps. This highlighted the need for optimization, even in basic projects, to maintain a smooth VR experience.
Takeaways
While Meta’s Quest Integration offers a solid foundation for VR development, the lack of updated documentation can be a barrier. For those starting out, begin with simple projects and utilize community resources extensively. Despite the challenges, the potential for creating immersive VR experiences is exciting and worth the effort.
Welcome to my journey into the fascinating world of Virtual Reality (VR) and Mixed Reality (MR) game development! This semester, I’m diving deep into understanding how to create visually stunning and highly performant VR/MR games that can run smoothly on standalone headsets like the Oculus Quest 3. In this first blog entry, I’ll introduce you to my research topic, discuss the unique challenges developers face, and explore why VR games vary so much in visual fidelity and user experience.
Why Focus on Standalone VR/MR Headsets?
Standalone VR/MR headsets are revolutionizing the way we experience virtual environments. Unlike tethered headsets, which rely on powerful PCs or consoles, standalone devices like the Oculus Quest 2 offer untethered freedom and accessibility. This opens up VR to a broader audience, making it crucial for developers to optimize their games for these platforms. However, creating games that are both beautiful and performant on standalone headsets presents unique challenges.
The Challenge of Balancing Performance and Visual Fidelity:
VR/MR games differ significantly in visual fidelity and user experience due to a variety of factors. Here are some of the key reasons:
Hardware Limitations: Standalone headsets have limited processing power and memory compared to high-end PCs. Developers must optimize their games to run smoothly within these constraints, which often means sacrificing some visual detail to maintain performance.
Rendering Techniques: Advanced rendering techniques used in PC-based VR games, such as real-time ray tracing, are often too resource-intensive for standalone devices. Developers must rely on more efficient methods like Level of Detail (LOD) and texture compression to achieve good performance.
Optimization Strategies: Effective optimization involves a mix of techniques, including occlusion culling (not rendering objects that are out of view), efficient shaders, and baked lighting (pre-computed lighting effects). These strategies help maintain high frame rates, which are crucial for a smooth VR experience.
Art Style Choices: The choice of art style can significantly impact both performance and visual appeal. Stylized graphics with simplified textures and models can look great while being less demanding on the hardware, whereas hyper-realistic graphics require more resources and careful optimization.
Why Do VR Games Differ So Much in Visual Fidelity and User Experience?
Red Matter 2, a rather optimised game with high fidelity graphicsPavlov Shack. The Quest 3 Standalone Port looks significantly worse than its PC counterpart.
The variance in visual fidelity and user experience across VR games can be attributed to several factors:
Developer Priorities: Some developers prioritize visual fidelity, creating stunning environments but potentially compromising on performance. Others focus on smooth performance, resulting in simpler visuals.
Budget and Resources: Indie developers might not have access to the same level of resources and optimization tools as large studios, leading to differences in game quality.
Target Audience: Games aimed at a broader audience might prioritize accessibility and comfort, opting for less intensive graphics to ensure smooth gameplay for everyone.
Technological Evolution: As standalone VR technology evolves, newer games can leverage more advanced features and optimizations, leading to disparities between older and newer titles.
Conclusion:
As I embark on this research project, my goal is to uncover and document the best practices for balancing performance and aesthetics in standalone VR/MR games. Through a combination of literature review, case studies, and hands-on development, I aim to create a prototype game that demonstrates these principles in action. Stay tuned as I explore the technical and creative aspects of this exciting field, and share insights and discoveries along the way.
Thank you for joining me on this journey. In the next entry, I’ll delve into the initial research phase, exploring existing optimization techniques and visual enhancement strategies used in the industry. Let’s make VR/MR games not only playable but truly breathtaking!
To explore one of my possible research topics further, this blogpost will look into the strengths and weaknesses of virtual, augmented, mixed and extended reality systems as well as their exact definitions and current technological trends, to gain a better understanding of which system / technology to use in future endeavors when going for different experiences.
VR vs AR vs XR/MR – a comparison
To better understand the difference and similarities between the different technologies, it is first important to understand their definitions as well as their strengths and weaknesses. For this reason, a short overview will be provided.
Virtual Reality (VR)
Virtual reality creates a computer-generated environment that immerses the user in a completely digital, three-dimensional space, often experienced through specialized VR headsets, providing a sense of presence and interaction within the virtual world. The user, thereby, is completely seperated from the real world and any stimuli they experience is completely computer generated. As a result, it usually allows for a much deeper immersion than the other solutions researched in this blogpost, but has its own strengths and weaknesses to be aware of.
Strengths:
Offers a completely immersive experience, perfect for training, gaming and simulations
Can easily create spaces, that are not accessible normally and / or provide space where normally there would be none
Can be used in healthcare, especially in therapeutic applications, to provide immersive therapy, exposure therapy, pain management and rehabilitation
Weaknesses:
Isolation from real world may cause emotional distress (solitude)
Needs special equipment, that may be costly and / or not readily available
Augmented Reality (AR)
Augmented reality overlays digitally created content onto our real world, enhancing the user’s perception of their surroundings by integrating computer-generated information such as images, text, or 3D models into the real-time environment. These are typically viewed through devices like AR glasses, tablets or smartphone screens, though in recent years, more applications have surfaced. Especially in the automotive industry, heads-up displays also make use of AR to display necessary information directly to the driver while projecting said information directly onto the windshield.
Strengths:
Real world information overlay, that relays information in real-time and provides additional input
Allows for hands-free interaction, which allows the user to engage the digital content while also staying aware in the real world
Useful for product visualisation and trying out products before making a buying decision
Weaknesses:
Limited field of view, especially on smartphone screens or tablets
Mobile dependency means less computing power, limiting display performance and causes need for optimisation
Mixed Reality (MR)
Mixed reality combines elements of both virtual and augmented reality, allowing digital and physical objects to coexist and interact in real-time. It seamlessly blends the virtual and real worlds, and allows switching between them, enabling users to engage with both types of content simultaneously. While this, of course, can provide difficult to understand with at first, it also allows for a much deeper influence of the user’s perceived reality as a result.
Strengths:
A high level of versatility, as it combines both VR and AR, it allows for a broader range of experiences to be created
Enables both in-room and virtual connection, communication and collaboration
Can, same as VR, be used in a wide variety of industries for training purposes, while also allowing for direct testing in the real world in AR
Weaknesses:
Different technologies and their implementation can cause performance and optimisation issues, posing technical difficulties
Cost of adaption currently still very high, especially when compared to pure VR or AR solutions
Extended Reality (XR)
Finally, extended reality is an umbrella term, encompassing VR, AR, and MR. It referes to the spectrum of immersive technologies, that extend, enhance, or blend reality. XR is a comprehensive term covering the entire range of experiences, from completely virtual to fully augmented. It aims to offer a holistic approach to immersive technologies. As such, it comes with all of the previously strengths and can, if used correctly, mitigate some of their weaknesses. The opposite, when used incorrectly, is also true, however.
Current trends and technologies
While of course, the market around VR, AR, MR and other technological solutions like that is constantly evolving, it is still important to understand the general direction of these developments in order to better understand them and work with them. As such, some of the most important ones will be listed here.
Stand-alone, wireless VR, AR and MR headsets without the need for external trackers:
HTC recently presented their new inside-out tracker, that would allow for inbuilt-tracking on a multitude of existing 3rd party headsets
While some of the previous generation’s systems still require for external trackers / tracking stations to be set up or for the headset to be connected to a PC via cable in order to ensure a smooth and immersive experience, current trends have since begun to diverge from that. The current trend seems to be to offer stand-alone, wireless VR, AR and MR solutions without the need for a seperate computer or similar unit for calculations, that also offers tracking via inbuilt sensors instead of external stations. The tracker shown in the picture above, for example allows for complete controllerless tracking of a person’s arms, legs, torso and even head, all by attaching it to the limb in question and up to 8 trackers can currently be used together to provide a smooth and easy experience.
Hybrid systems:
Both the Apple Vision Pro and the Meta Quest 3 offer hybrid solutions when it comes to VR and AR.
While there are still certainly solutions, that focus on either VR or AR in particular, MR seems to be the much more common trend, with passthrough headsets becoming more and more widespread. This is especially obvious when looking at both Apple’s recently released Vision Pro, but also Meta’s Quest 3.
Wider application in everyday life:
When looking at the adaption rate of VR, AR and MR solutions, it quickly becomes apparent that the possible applications for it have skyrocketed. While originally more of a niche development, nowadays a wide variety of experiences are offered, ranging from immersive nature documentaries, sport events, games, movies and more. Furthermore, with both the increasing variety of headsets to choose from and their increasing calculation power comes a lower price – except for the apple solution, of course – with an also increased quality in the experience, making the technology much more accessible in everyday life.
Next steps:
Look further into different XR solutions and their respective issues
Research essential tools for creating immersive virtual environments
Check methods of engagement and interaction within these digital environments
XR is a technology that has been gaining popularity in the entertainment industry. While gaming is a major part of XR, this post would explore its other applications in entertainment, such as virtual concerts, immersive theater, and interactive art installations.
Virtual Concerts: A New Stage for Artists and Fans
Imagine attending a concert by your favorite artist from the comfort of your living room, yet feeling as though you’re right there in the front row. XR makes this possible. Virtual concerts in XR are not just about streaming a live performance; they are about creating an immersive, interactive experience. Fans can choose different viewpoints, interact with the environment, and even feel the vibration of the music through haptic feedback technology.
Artists like Travis Scott and Marshmello have already experimented with these concepts, drawing millions of virtual attendees. These events aren’t just concerts; they’re hyper-realistic experiences blending music, visual art, and digital interaction.
Meta is also pushing strongly into this direction by hosting live concerts on their Meta Quest Plattform. There will be for example a Lice concert by imagine Dragon at June 15th on this plattform.
Immersive Theater: Blurring the Lines Between Audience and Performer
Theater has always been an immersive experience, but XR takes this immersion to a new level. Unlike traditional theater, where the audience is a passive observer, XR theater can make viewers a part of the performance. Through VR headsets or AR applications, audience members can experience different narratives from multiple perspectives, interact with the performers, or even influence the outcome of the story.
Companies like Punchdrunk and Magic Leap are pioneering in this space, creating experiences where the line between audience and performer is blurred, leading to a more intimate and personal form of storytelling.
Interactive Art Installations: Stepping into the Canvas
Art has always been a medium for expression and experience, but XR adds an interactive dimension to it. Interactive art installations using XR technologies allow viewers to step into the artwork, manipulate elements, and experience the art in a multi-sensory manner. This form of art is not just to be seen but to be experienced and interacted with.
Artists like Refik Anadol and teamLab are at the forefront, creating stunning visual landscapes that respond to and evolve with viewer interactions. These installations are not static; they are dynamic and alive, offering a personalized experience to every viewer.
Conclusion: A New Era of Entertainment
XR in entertainment is more than a technological advancement; it’s a paradigm shift in how we experience art, music, and storytelling. It’s about creating worlds that we can step into, interact with, and be a part of. As we look to the future, the possibilities are boundless. We’re not just witnessing a change in entertainment; we’re participating in the birth of entirely new forms of expression and experience.
This is just the beginning. As XR technologies continue to evolve, we can expect to see even more innovative and immersive experiences that challenge our perceptions of reality and entertainment. The future of entertainment is here, and it’s virtual, augmented, and mixed.
Environmental Interface: Designers must consider the entire surrounding environment as a potential interface, moving beyond the confines of a flat screen.
Comfortable Focusing Range: Interactive elements should be placed within a range of half a meter to 20 meters, the comfortable focusing distance for human eyes.
Beyond Reach: For interacting with objects 20 meters away, MR utilizes tools like handheld controllers or technologies such as eye tracking and hand recognition.
Eye Movement
The human eye comfortably moves 30°-35° in all directions, creating a field of view (FoV) of about 60°. Key UI elements should be placed within this range for easy accessibility.
Key Elements are arranged in a FoV of ~60°
Arms Reach
The average arm’s length is 50–70 cm. Essential interactions should be positioned at this distance for ease of use.
Designing for Distance
Drawing from Kharis O’Connell’s “Designing for Mixed Reality”, the interaction space is divided into three layers:
Interaction Plane: Core UI elements are placed within arm’s reach.
Mid-Zone: For placement of virtual objects in MR.
Legibility Horizon: The limit for comfortable focus and reading, approximately 20 meters. Beyond this, only images should be used.
Addressing User Fatigue
Ease of Exit: Always provide a straightforward method to exit or pause, like a button.
Save Functionality: Allow users to save progress to prevent data loss and alleviate exit anxiety.
Scaling and Interaction
Button Size: Ensure buttons are large enough, with a minimum size of 2 centimeters.
Natural Interactions: Mimic real-world interactions, like picking up a mug by its handle.
Poses and Gestures
Clear Instructions: Given the novelty of MR, provide explicit instructions for poses and gestures.
Simplicity: Use poses and gestures sparingly to avoid overwhelming users.
Feedback and Guidance
System Feedback: Implement feedback mechanisms like haptic feedback or color changes when interacting with virtual elements.
Clear Guidance: Offer concise and clear instructions, crucial in the unfamiliar terrain of MR.
Mixed Reality is not just a new technology; it’s a new way of interacting with our world. As we design for MR, we must consider the unique physical and perceptual aspects of this medium. By focusing on intuitive interactions, comfortable viewing distances, and clear instructions, we can create MR experiences that are not only engaging but also accessible and user-friendly. The future of MR is bright, and as designers and technologists, it’s our responsibility to pave the way for this exciting new era of digital interaction.
