[StarDesk] Develop a virtual desktop adapter for multiple virtual displays

Hey AVP Community, I spent the bulk of my weekend attempting to get as low latency gaming on the avp as possible. I think before I list everything I try it is fair to mention that in most of these apps cases they are limited by cloud rerouting, poor compression settings, or wifi bandwidth limitations. Lets start with where I began: * Xbox App * Worked horribly. From some googling and AI research I came to understand that the onboard compression from xbox to streaming is noticeably bad since it isn't built for low latency realtime streaming over lan but more cloud networking. * Steam Link * Just Ok. When connecting to a pc to play games on it works alright. Definitely could play an offline story mode game with zero overhead and no cost, but its not the solution im looking for. While the xbox app sucked and steam link was ok it helped me understand that the issue lied in bandwidth and transfer speeds to the AVP over wifi. My router setup isn't known or configurable based on where I live, but I can get about 500mbps/20mbps download/upload. Now for my setup I want 4k HDR. I want to be able to enjoy all the commodities of gaming on a headset that allows me to game anywhere. Which led me down the rabbit hole of building my own solution. This solution wouldn't depend on an avp connection to the streaming device. I found this out the hard way. There is no good way to route 4k to the avp over wifi with a max of 20mbps upload without hella encoding and decoding on both sides (more latency). SO I just decided to let virtual desktop handle the handoff since it is proprietary apple comms and works perfectly when doing work, so why not. I have used an elgato game capture card before but never in this way. I started by attempting to serve a streamed hd video over a udp serving connection over wifi to my Mac and had it decoded using ffmpeg. I was trying to do everything I could to ensure that we had low latency between my capture card on my pc and the stream on my Mac which would be mirrored into the avp. * Elgato Game Capture Setup * UDP serving to Mac over wifi * Horrible bandwidth issues * Bad compression quality and high loss of packet data * horrible user setup (don't try it unless you know computers) * Ninja.OBS * Streams to Safari on avp * Better than every other solution above * Still some frame loss and not realistic for realtime gaming The only issue is that the capture card I was using is from like 2010 so the hardware latency encoding is 150-200ms on top of a 10ms data transfer at like 20 mbps transfer speeds (nearly impossible without a couple seconds of buffering). So I decided to just 1gb ethernet the streaming pc and the Mac together. This was done by taking the video input from the capture card and routing it over the ethernet connection with sub 1ms latency packet pining on avg (down 9ms avg from wifi) using NDI solutions. Literally a game changer, with tech from 2010. The video posted with this is a direct run of this on marvel rivals. [Remote Desktop AVP Gameplay](https://reddit.com/link/1jt662y/video/6l1c4u8fdate1/player) This solution works fine and you can see visually there is still some latency. I tried playing some rocket league and cod through this medium and it was ok, but way easier to lose since the latency here is like 0.6s. So it is not a solution yet, but it led me to my next stage of testing. Now the only thing holding me back are encoding and transfer speeds so I bit the bullet and decided to just buy the 4k x from Elgato. This removes the streaming pc and the 1gb bottleneck. It'll work as a low latency solution to directly path from xbox to Mac with Elgato's Instant Gameview (maybe with some upscaling) with 5gb of bandwidth based on the usb 3.2. It hasn't come in yet, but if I need realtime hardware decoding from it then I can since it outputs h.264/ hvec file streams for passthrough which apple devices are designed to decode with hardware (low latency). I haven't gotten the 4k x yet, but I'll post an update once I try it out if you guys like this project. Also, this is kind of a tangent since it feels like apple is stuck with just the Remote Desktop from a single Mac. I feel like this product needs some kind of virtual desktop adapter so you can have multiple virtual displays in realtime up. It would make this whole process easier and could be done with a Mac mini and a 4k60 mk2 from Elgato. That way I don't need to purchase a capture card and a Mac, just a single $500 unit. And before anyone gets upset, Mac Virtual Display feels lossless and works extremely well, I just wish they'd extend it while ensuring quality. LMK what you guys think, thanks.