Guide for using Apple Vision Pro as HDMI display

HDMI on Apple Vision Pro.md

Displaying HDMI sources on Apple Vision Pro

While it's possible to stream most content to Apple Vision Pro directly over the internet, having the ability to use Apple Vision Pro as an HDMI display can still be useful.

Since Apple Vision Pro does not support connecting to an HDMI input directly or using an HDMI capture card, we have to be a little creative to make this work. NDI provides the ability to stream HDMI content over a local network with really low latency, and it works great with Apple Vision Pro.

This page shows the setup I’m using.

What’s needed

• HDMI NDI Encoder
  • Personally, I recommend the Kiloview N40, as it supports streaming up to 4K60 while being fan-less, and can use USB-C as a power input.
  • Kiloview N60 comes with more features, but it’s larger and has a fan.
• Vxio app
  • I made this one, you can also use other NDI monitor apps as well.
  • Separately, Finn made Castaway that makes it possible to use a USB Capture Card + Mac/iPad instead of an NDI encoder. It's a more cost effective option for most people I imagine.

Things to consider

• To ensure high video quality, an NDI stream typically consumes significantly more bandwidth than a typical H.264/H.265 stream. Before purchasing the encoder, make sure your network can support data transfers at that speed.
  • For 4K60 stream, the bandwidth required is typically around 250~300mbps. If you're having issue to stream smoothly, try go to the encoder's settings page and reduce the streaming quality.
  • You can test this with some free software NDI encoders.
• Since this setup requires encode and decode the video content, it is not latency free. From my experience, I typically see 3~4 frames (~50ms, same as moonlight based on my measurement) of delay from the content source.

Setup

Setting up an NDI encoder is pretty straightforward. Just connect the device to your local network via Ethernet, supply power, and connect the HDMI input to the encoder.

If the visionOS shows the Local Network permission dialog, and after granting the permission the app still shows searching, you may need to force close the app, and reopen it.

After that, you should be able to see the encoder show up as a source in the Vxio app. Select the source, and you'll get your HDMI display 🎉

Misc

Adjusting audio queue size

Depending on the NDI encoder, the default audio queue size might be too small for some devices. In that situation, try increasing the audio queue size so that the audio no longer experiences abrupt stops between samples.

Disable Multi-TCP

If your Wi-Fi channel is congested, you can try force N40 to use UDP instead of Multi-TCP. In my experience, UDP performs a lot better in those network condition. You can do so from the N40's configuration UI, select NDI Connection -> Multi-TCP Disallowed.

Portable setup

I was able to make this setup portable by getting one of those mini router (GL.iNet Beryl AX) with a LAN port.

#AppleVisionPro #visionOS

@lightsailvr The side by side mode adopts the aspect ratio of the underlying video source. If you want the 1:1 ratio, try adjust the encoding side so the width and height are equal.

@KhaosT When I use the suggested solution for the latest N60 firmware HDR issue, the color override does fix the greyish colors, but the image gets very blurry, when using the color override. Is there a way to fix this?

+1 on @LowRiderXR's comment above (I have also observed a slight blurriness when applying the color override)

@KhaosT also a feature request to support Apple Vision Pro environments in the AVP app 🙏 would be awesome to play PS5 games reflecting on a lake

but the image gets very blurry, when using the color override

I believe visionOS 2.2 broke the HDR video display for content shown using AVSampleBufferDisplayLayer since the same code works fine on iPadOS, Apple needs to fix that.

a feature request to support Apple Vision Pro environments in the AVP app

Currently Apple only provided API to let app do this for normal video content playback using AVPlayerViewController, which only works with HLS content over internet. There is no way for the app to do it using the display layer.

So the greyish tone of the HDR is not a kiloview firmware issue, but rather a visionOS bug? Is it possible to report the bug to apple?

So the greyish tone of the HDR is not a kiloview firmware issue, but rather a visionOS bug?
No, there are two things.

1. N60 firmware isn't sending the NDI HDR metadata correctly right now. The color override fixes that.
2. visionOS 2.2 displays HDR content incorrectly (blurry) when using AVSampleBufferDisplayLayer. If you take out the color override, the content should be pretty sharp since HDR is not triggered.

Is there a conversation with Kiloview to get the HDR metadata set correctly? I've found them to be pretty responsive at [email protected], but I myself probably don't have the know-how to explain/verify the issue

@haseebrabbani yeah the Kiloview thing they are aware of that since end of November. They actually sent out a test firmware that have the correct metadata so initially I didn't add the color override feature. I was a little surprised that the final released version didn't include their fix, not sure what happened.

anyway you could provide a link to the test firmware? 😁

@haseebrabbani Unfortunately no. It's not much different from the release version, only the HDR metadata thing which you can correct with the color override thing. The visionOS 2.2 issue happens regardless whether it's showing the HDR content based on the accurate metadata, or from the override.

got it. so is there a conversation/ticket with Apple to fix this? if not, I can help with that. I'll just go ahead and file that in their Feedback app anyway so theres more voices

I didn't file a radar since they never fix my radars from the past and it seems like a waste of time 🙃 My hope is just some high profile company also running into the issue and then Apple will fix the problem for them 😛

no worries, just filed in their Feedback app.

also, can we get a tvOS app as well? would be nice to have for my setup

@KhaosT
I‘ve written to the kiloview support. They claim that the ndi HB meta data is correct, but there seems to be an anomaly in the ndi HX metadata, which they still try to fix.

Can I use the ndi HX mode with the vxio app or only the the HB mode? When I use the HX mode, I get the sound, but only a black screen.

I think when HX mode only works if you set the N60 to only do HX. At least for me, when both HX and HB mode is enabled, I also get a blank screen for HX.

Thank you, I tried it yesterday. HX mode does work, HX3 seems not to work. I will wait for the visionOS fix.

hey @KhaosT 🙂 been thinking about a fix to the blurry HDR issue. I noticed that when I am resizing the VisionOS window from the bottom left/right corner the image appears sharp. When I release the resizing bar then the image goes back to being slightly blurry. Could this be used as a "workaround" to fix the blur? (i.e. programmatically resize the window by like a pixel and repeatedly make it larger and smaller so that its hopefully not noticeable)

another thing I wanted to point out is that sometimes the HDR works without blur. i dont have a solid repro, but on occasion the Vxio app seems to freeze/crash and then when i restart it I can see the stream in HDR without blur (so glorious!). there must be a way to fix the blur consistently

I’ll see if there are some workaround I can apply to fix the issue this weekend. Basically the problem is we passed in the 4K HDR buffer to visionOS and somehow it’s foveated rendering pipeline decided it won’t show that in full resolution…

@haseebrabbani after isolating the issue, the end result is AVSampleBufferDisplayLayer + 4K HDR sample buffer. If I drop the HDR tag it works, and resize the app window also forces the OS to redraw the blurry frame. Unfortunately, I wasn't able to find a workaround as programmatically resizing the app window won't do that.

Appreciate you looking into it 🙏

is anyone aware of other NDI monitor apps for AVP that work with non-blurry HDR? More than happy to pay for scratching this itch

@haseebrabbani in the next update there is going to have an option to use RealityKit to render the stream, which seems to work with HDR content. Unfortunately based on my testing RealityKit has a different set of bugs which prevents it working for non-HDR content. You might switch between them based on what content you're trying to view.

@KhaosT that would be amazing! as long as there is some option to make the HDR appear non-blurry, that would be much appreciated 🙏💪 is there an ETA on the next update?

It’s already out

GLORIOUS! thank you 🙏

Thank you @KhaosT for the continued updates! I just tried viewing HDR content in immersive stereo, but the highlights are sitll way too flat (almost like they are double tone-mapped). You can see the difference pretty clearly if you open the stream in "regular" flat mode in Vxio then enter immersive, as you'll see the flat version floating atop the immersive space. The regular version looks wonderfully colorful and bright in P3 ST2084 but the immersive situation is still very dull. The saturation seems correct, however. I also noticed that switching between the two renderers (Reality Kit and the AVP native) made no difference, at least not in immersive mode. I tried toggling this setting while in immersive and also by selecting one or the other, force quitting VXIO, then coming back in. I was a little thrown off by the fact that not check mark appears next to which option you select, which made me wonder if my selection was indeed taking hold.

@Jefe533 oh immersive one didn't change because it's a different pipeline and since visionOS refuse to support standard SBS setup, supporting HDR in that is going to be super annoying so I didn't bother.

That's too bad but I understand.