At Zappar we’ve been living and breathing immersive technologies for over 13 years bringing the most accessible tech solutions to the broadest range of people as possible. And so, as you may imagine, we’ve been itching to get our hands on Apple’s mixed reality headset, Vision Pro.
Apple provides several ways for building immersive experiences for Vision Pro, including Unity and Xcode, but the one we’re particularly interested in at Zappar is WebXR. Experiences built for WebXR are distributed over the web - accessed through the web browsers included with headsets - and that’s a game changer for a number of reasons:
- For those building or publishing XR experiences it means no App Store submission or review processes, and no need to enrol devices in developer mode. Content can be built once and viewed on a number of different headsets without modification (more on this below!). It’s even possible to leverage some of the great tooling that exists in the frontend web development ecosystem (including Mattercraft!).
- For end users accessing XR experiences, it means there’s no need to install an app for each piece of content, making it super fast and easy to get straight into the action.
In this blog, we’ll take a technical deep dive into Vision Pro’s support for WebXR. We’ll see that Apple Vision Pro is a great device for experiencing interactive WebXR content, but there are some important implementation details that make Vision Pro a bit different from other VR/MR headsets.
We’ll cover:
- A quick primer on WebXR
- How to enable WebXR support on Vision Pro
- Vision Pro’s support for the various immersive modes offered by the WebXR specification
- Hand tracking and user input
- How to share and launch WebXR experiences on Vision Pro
- And finally, some comments on developer experience when building content for Vision Pro.
Let’s get started!
What is WebXR?
WebXR is a set of standards that allow web pages to offer immersive 3D content, including virtual reality, augmented reality, and mixed reality experiences. These standards are implemented (to varying degrees of completeness) by the web browsers included with VR/MR headsets, such as the Meta Quest 1/2/3/Pro, the Magic Leap 2 and (as we’ll see) Vision Pro.
The promise of WebXR is simple, build content once, targeting the WebXR specification, and have it run without modification on these different headsets. While that’s true in principle, in practice it’s somewhat complicated by the degree of implementation of the standards by web browsers, and the differences in form factor and input mechanisms (e.g. handheld controllers versus hand tracking) of the various headsets. As we’ll see throughout this deep dive, the Vision Pro is a relatively unique device, with corresponding opportunities and challenges for delivering XR content over the web.
In many cases it’s possible for developer tools to address these differences in implementation, allowing content creators to build experiences that run consistently across a wide range of devices. That’s exactly what Mattercraft does - it’s our fully featured development environment for building 3D web-based experiences, and its WebXR support provides a range of abstractions that make building rich and interactive content for headsets a breeze.
Enabling WebXR Support on Vision Pro
Apple considers the WebXR support on Vision Pro to be ‘experimental’ and requires the user to enable it in the Safari settings:
From the Settings app, head to Apps, then Safari, Advanced, and Feature Flags then enable the WebXR Device API and WebXR Hand Input Module. It’s then best to restart Safari to ensure that any pages are reloaded with the correct features enabled.
While this isn’t ideal for end users, it’s a process that only needs to be completed once. After it’s in place, a prompt to enter ‘Immersive Mode’ will be presented to the user whenever they visit a web page in Safari that requests a WebXR session.
Supported WebXR Modes
The WebXR specification supports two primary modes of operation:
- immersive-vr where an experience presents a fully virtual environment to the user; and
- immersive-ar where the device shows the user’s real-world environment and digital content is added into that space (typically known as mixed reality, or augmented reality)
While Vision Pro is conceptually a ‘mixed reality’ device, showing the user their real-world environment, its WebXR implementation currently only supports the immersive-vr mode. This means that experiences must provide a background or virtual 3D environment.
In Mattercraft, we’ve added the ability to choose a primary WebXR mode and a ‘fallback’ mode for a given project. That means you can deploy experiences that run in mixed reality on devices that support it (such as Quest, Magic Leap, and our Zapbox headset) but still present a working (fully virtual) user experience on devices that don’t (like Vision Pro).
Hand Tracking and User Input
The WebXR specification provides several ways for users to interact with content, including:
- handheld controllers, such as those included with the Quest or Magic Leap headsets. These typically include a number of buttons, and may include a thumbstick or touchpad.
- hand tracking, where experiences can react to the positions of the user’s hands in 3D space, including the poses of their fingers and thumbs.
To simplify the development of interactive WebXR experiences, Mattercraft provides a feature called ‘pointer emulation’. With it, developers build experiences using the point-and-click events that are common when building a traditional web page (e.g. ‘click’, ‘pointerdown’, ‘pointerup’). Mattercraft automatically maps the input mechanisms (either controllers or hands) exposed by any given headset to these events using rays in 3D space.
Vision Pro is not shipped with any controllers. Instead, users interact with its interface and apps using an ‘eye gaze plus finger/thumb pinch’ mechanism. While this feels intuitive and somewhat natural, Apple has unfortunately not exposed it through their WebXR implementation. We believe they’re working to influence the specification going forward to include abstractions for this kind of input, so we’re hopeful that this may implemented in the future.
There’s one other important feature of hand-tracking implementations to note - the two ‘anchors’ typically associated with a given hand. These are points in space that the device tracks the position of and makes available to WebXR content. They are:
- target ray space, which is located just beyond the hand and aligned with the direction that the user’s hand is naturally pointing towards; and
- grip space, which is positioned in a consistent location relative to the hand where the user would hold an object
Of this pair, Vision Pro implements only target ray space. This is not wholly unexpected - grip space is optional in the WebXR specification - so, to make things easier for content developers, Mattercraft emulates grip space using a point in space offset from the wrist location reported by Safari.
Sharing and Launching WebXR Experiences
Once you’ve built and deployed a WebXR experience, the next step is to get users trying it out! Since WebXR experiences are web pages, distributing your content is as simple as distributing a web link. Vision Pro has a number of mechanisms that may be of use in this regard:
- Users can launch an experience by clicking on its web link in Vision Pro’s Safari browser, Mail app, Messages app, or indeed in any other social app they have installed.
- Web links can be shared to a Vision Pro headset from a Mac or iOS device using AirPlay
- Web links copied on a Mac or iOS device can be pasted into Safari on Vision Pro using the Universal Clipboard feature of the Handoff function
- Web links copied on a Mac that’s being controlled using Vision Pro’s ‘Mac Virtual Display’ feature can be pasted directly into Safari
Some other headsets support scanning QR codes, but unfortunately Vision Pro does not, neither with the built-in Camera app, nor in apps from the App Store. We believe this is unlikely to change in the near future as Vision Pro does not expose the external camera to apps or web pages.
Developer Experience
It’s hugely important, when building for any platform, that the process for developers is comfortable, productive and efficient. A great developer experience is one that lets the developer spend the most time thinking about what they’re trying to achieve and less about how they’re going to achieve it. In this regard, Vision Pro has some great features that set it apart from other headsets.
The most prominent of these features is ‘Mac Virtual Display’. While wearing a Vision Pro, looking at a ‘real life’ Mac in your immediate area shows you a ‘Connect’ button. Selecting this opens a large virtual screen for that Mac in front of you that you can use to operate without having to remove the headset. With this feature, you can build WebXR content on the virtual Mac using your existing development workflow and tooling, while seamlessly testing and previewing that content in the Vision Pro browser. It’s perhaps easier to show than describe, so we’ve recorded a short video demonstrating it.
Needless to say, we’ve made sure that Mattercraft’s Live Preview feature works great with this workflow. It makes for an almost instantaneous iteration time between change and testing when developing an XR experience.
Also of note with regards to developer experience is that the Vision Pro is relatively comfortable to wear for moderate amounts of time, particularly if using the ‘dual loop’ alternative headstrap included in the box. While the ‘Virtual Display’ doesn’t quite reach the quality of a real-life monitor, text (and code) is still sufficiently crisp to be read and written with ease.
Wrap Up
As we can see, even with the ‘experimental’ nature of the headset’s WebXR support, it’s possible to build high-quality, interactive XR experiences for Vision Pro for deployment over the web. The quality of the headset itself, along with some great features for developers, mean it’s a pleasure to build content with and for.
That said, it’s clear that other headsets, such as Quest and Zapbox, have fantastic overall value propositions, including price point, user input options (aka controllers), and maturity/breadth of support for the WebXR specification. The beauty of the cross-platform nature of WebXR, in tandem with great tooling, is that you don’t have to put all your device eggs in one basket!
If you have any questions or comments regarding WebXR, Vision Pro, or Mattercraft then we’d love to hear from you! Feel free to drop into our Discord server, or drop our award-winning customer success team a message.
How to get started
Start building immersive web experiences for Vision Pro today with a free 14-day trial of Mattercraft.
Connell Gauld