Apple Vision Pro — Doubt Reality

On June 5th, 2023, after years of waiting, Apple finally revealed Apple Vision Pro to the world.

On paper, some might argue that the Apple Vision Pro is the ultimate VR headset. It has near-perfect passthrough capabilities, AR support, super high resolution displays, a 3D display on the outside which solves the “I can’t see your face” problem, high-res 3D cameras for capturing reality, a laptop-class processor, and the most beautiful VR headset design to date. The only issues are the two-piece design with tethered 2-hour battery pack and the lack of motion controller support.

That last one is big: that means less than 1% of existing VR games could be ported to it easily in theory, and even less than that will go through the effort to do so. That means that most content for the device are either going to be traditional 2D applications floating in your real-world space, or immersive experiences designed exclusively for the device.

But is that a bad thing?

You can communicate with your hands during a FaceTime call

Maybe this paradigm will be a much-needed reset for the VR industry. Apple refuses to use the term “VR”, but surprisingly has no issues with AR or 3D.

Speaking of 3D, how does that work? Well, Apple has defined their own method of how to construct 3D videos in a little-known format, MV-HEVC, to work on their headsets with some interesting capabilities, including the ability to dynamically move subtitles and closed captions closer to the viewer as the depth of the objects in the video underneath the subtitles or closed captions come further out of the screen. It’s not a novel idea, but it is a novel solution and implementation. Their tools will automatically generate a disparity map from your 3D video during the encoding process which will be used by their media player SDK to reposition the text boxes dynamically. Once you’ve authored a 3D video in their format, you could play it locally or set it up to stream using HLS. And of course you’ll be able to sell 3D movies you’ve created through the iTunes Movie Store too. Movies can be 24fps which will run the display in a special 96hz mode, or they can be up to 4K 3D HDR at 90fps with a high bitrate. Truly an incredible capability, something not seen in any other VR device so far. And Avatar 2 will be available in 3D on the device, delivered through the iTunes Movie Store.

A still from the Avatar: The Way of Water

What about other content though? Well, it’ll be able to mirror content from your iOS and MacOS devices, and will be able to run at least iOS applications locally and possibly MacOS apps too, due to it’s powerful M2 chip.

Can the cameras do anything cool besides the standard VR stuff like 6dof head tracking and hand tracking? Yes! Apple has just released new information about their Object Capture API and Object Capture sample project. Now, the photogrammetric reconstruction can happen locally on iOS devices, and because it’s part of RealityKit, that should include Apple Vision Pro. This means when Apple Vision Pro comes out, you’ll be able to capture reality with your face, just by looking at the thing you want to capture and walking around it.

A marketing image demoing a Spatial Video capture

Though on the surface, Apple Vision Pro may seem just like a slightly better VR headset like others we’ve seen before, but when you dig into the capabilities and tooling highlighted at WWDC, it’s clear that this is an XR device like we’ve never seen before. An XR device that’s an actual, honest-to-god, general purpose 3D computer. Or you can call it a “spatial computer” if you want to stay on-brand.