I mean a wireless wearable computer would have network lag but the important lag – between the rendering GPU and the screen – would be nil.
I mean streaming the video data to the headset Miracast-style would be dumb, I agree, but afaik all the rendering hardware in the new Apple headset is inside the headset.
The new Steam Deck and its copycats have proven that a compact gaming-grade PC is doable. If not fully contained in the headset, then make a little fannypack with a cable up to vr headset, but you can walk around untethered otherwise.
Headsets in the thousand-dollar range are plenty good and still not selling. Take the hint. Push costs down. Cut out everything that is not strictly necessary. Less Switch, more Game Boy.
6DOF inside-out tracking is required, but you can get that from one camera and an orientation sensor. Is it easy? Nope. Is it tractable for any of the companies already making headsets? Yes, obviously. People want pick-up-and-go immersion. Lighthouses were infrastructure and Cardboard was not immersive. Proper tracking in 3D space has to Just Work.
Latency is intolerable. Visual quality, scene detail, shader complexity - these are nice back-of-the-box boasts. Instant response time is do-or-die. Some monocular 640x480 toy with rock-solid 1ms latency would feel more real than any ultrawide 4K pancake monstrosity that’s struggling to maintain 10ms.
Two innovations could make this painless.
One, complex lenses are a hack around flat lighting. Get rid of the LCD backlight and use one LED. This simplifies the ray diagram to be nearly trivial. Only the point light source needs to be far from the eye. The panel and its single lens can be right in your face. Or - each lens can be segmented. The pyramid shape of a distant point source gets smaller, and everything gets thinner. At some point the collection of tiny projectors looks like a lightfield, which is what we should pursue anyway.
Two, intermediate representation can guarantee high performance, even if the computer chokes. It is obviously trivial to throw a million colored dots at a screen. Dice up a finished frame into floating paint squares, and an absolute potato can still rotate, scale, and reproject that point-cloud, hundreds of times per second. But flat frames are meant for flat screens. Any movement at all reveals gaps behind everything. So: send point-cloud data, directly. Do “depth peeling.” Don’t do backface culling. Toss the headset a version of the scene that looks okay from anywhere inside a one-meter cube. If that takes longer for the computer to render and transmit… so what? The headset’s dinky chipset can show it more often than your godlike PC, because it’s just doing PS2-era rendering with microsecond-old head-tracking. The game could crash and you’d still be wandering through a frozen moment at 100, 200, 500 Hz.
I think the slightly more viable version of the rendering side is to send a combination of low res point cloud + low res large FOV frames for each eye with detailed (!) depth map + more detailed image streamed where the eye is focused (with movement prediction) + annotation on what features are static and which changed and where light sources are, allowing the headset to render the scene with low latency and continuously update the recieved frames based on movement with minimal noticable loss in detail, tracking stuff like shadows and making parallax with flawlessly even if the angle and position of the frame was a few degrees off.
Undoubtedly point-clouds can be beaten, and adding a single wide-FOV render is an efficient way to fill space “offscreen.” I’m just cautious about explaining this because it invites the most baffling rejections. At one point I tried explaining the separation of figuring out where stuff is, versus showing that location to you, using beads floating in a fluid simulation. Tracking the liquid and how things move within it is obviously full of computer-melting complexity. Rendering a dot, isn’t. And this brain case acted like I’d described simulating the entire ocean for free. As if the goal was plucking all future positions out of thin air, and not, y’know, remembering where it is, now.
The lowest-bullshit way is probably frustum slicing. Picture the camera surrounded by transparent spheres. Anything between two layers gets rendered onto the further one. This is more-or-less how “deep view video” works. (Worked?) Depth information can be used per-layer to create lumpen meshes or do parallax mapping. Whichever is cheaper at obscene framerates. Rendering with alpha is dirt cheap because it’s all sorted.
Point clouds (or even straight-up original geometry) might be better at nose-length distances. Separating moving parts is almost mandatory for anything attached to your hands. Using a wide-angle point render instead of doing a cube map is one of several hacks available since Fisheye Quake, and a great approach if you expect to replace things before the user can turn around.
But I do have to push back on active fake focus. Lightfields are better. Especially if we’re distilling the scene to be renderable in a hot millisecond, there’s no reason to motorize the optics and try guessing where your pupils are headed. Passive systems can provide genuine focal depth.
My suggestions are mostly about maintaining quality while limiting bandwidth requirements to the headset, wouldn’t a lightfield require a fair bit of bandwidth to keep updated?
(Another idea is to annotate moving objects with predicated trajectories)
Less than you might think, considering the small range perspectives involved. Rendering to a stack of layers or a grid of offsets technically counts. It is more information than simply transmitting a flat frame… but update rate isn’t do-or-die, if the headset itself handles perspective.
Optimizing for bandwidth would probably look more like depth-peeled layers with very approximate depth values. Maybe rendering objects independently to lumpy reliefs. The illusion only has to work for a fraction of a second, from about where you’re standing.
Alpha-blending is easy because, again, it is a set of sorted layers. The only real geometry is some crinkly concentric spheres. I wouldn’t necessarily hand-wave Silent Hill 2 levels of subtlety, with one static moment, but even uniform fog would be sliced-up along with everything else.
Reflections are handled as cutouts with stuff behind them. That part is a natural consequence of their focus on lightfield photography, but it could be faked somewhat directly by rendering. Or you could transmit environment maps and blend between those. Just remember the idea is to be orders of magnitude more efficient than rendering everything normally.
I thought the windows MR lineup filled that gap pretty well. Much cheaper than most of the other alternatives back then but it never really took off and MS has quietly dropped it.
Still $300 or $400 for a wonky platform. That’s priced better than I thought they were, but the minimum viable product is far below that, and we might need a minimal product, to improve adoption rates. The strictly necessary components could total tens of dollars… off the shelf.
He’s right, a product like that would have failed dramatically. At this point I just want them to release a dumb AF, streaming-only, inside-out tracking VR headset that connects to PCs. Forget trying to cram an expensive Qualcomm or AMD chip in there, it will never give you the ideal VR experience. Make something that’s $200 bucks, connects to any PC running SteamVR, and just does extremely well with streaming and low-latency. Both Airlink and VRDesktop have already shown that its possible to get extremely close to a cabled experience. All that’s left is some polish.
I just want to be able to buy something like an Oculus CV1 without Oculus software/proprietary hardware and a nicer screen. I’m still rocking the same unit I got several years ago and it’s still plenty fine for most things.
All of the fancy things like wireless and no-tower tracking are nice, but I imagine a lot of players are going to be seated and just want the immersion. Why not have a $300-400 offering that does this?
I could be wrong on this since I have no source but I always assumed that Oculus headsets were cheap because they’re a Meta product and you’re actually paying for it with your data/telemetry.
Like, the ungodly amount they spent on VR R&D is absolutely not being made up for by the few hundred dollar price tag on their headsets — I bet that barely covers the cost of materials. That must be for a reason.
I think having base stations not only increases price but also makes it unapproachable for a vast majority of people. Personally, I didn’t even consider the CV1 or the Index because I just didn’t have a room that could properly accommodate them. For the sitting use case, no-tower tracking is actually very suitable and probably works better.
even at ideal condition there is about 1~2ms latency(streaming 1080p game), while hitting 90hz requires 11ms frame time. so you are asking the game to at least perform at 111fps or above to function under said ideal condition. I think if some manufacturer can put together a chip set where they do the frame gen tech on the head set side, so the game just need to run at 60fps it would be a better option.(like PS VR ) Frame gen does require some other buffers to generate the in between frames, so that’s more info to stream over the bandwidth.
That’s basically what I got. Xreal Air (formerly Nreal until a C&D from Epic). 1080p per eye and something like 49PPD with a 45° FOV. Tracking is 6DOF and requires software on the host (only complaint) and connectivity is via a USB-C cable (uses DP alt mode).
It’s nearly as “dumb” as an HMD can get. From the teardowns that I’ve seen, it’s really just got an MCU, a GPIO expander, a 6DOF chip, and the displays + drivers. And I love that about it. No batteries or anything to worry about.
It is pretty narrow but also what makes it work, IMO. I don’t have them for immersion but for display replacement. The narrow field of view lets the 1080p display have nearly 0 screen door effect. Plus, the birdbath optics are really cheap compared to waveguides or fancy lenses in VR headsets.
This has been on my radar for a while to compliment my steam deck. But I believe it doesn’t do head tracking with the steam deck or does it? I just want a floating screen in front of me that stays still when moving my head around, otherwise I’m gonna hurl!
I think it’s great for my Deck but, that will indeed be a problem. The headset contains only the sensors and display systems but, none of the logic circuitry to “pin” displays. Including that would increase the price a good deal.
Understood thanks for the feedback. After posting you reignited my interest and I found out that they also have their product called beam which would do the trick to make a spatial display… if you’re willing to cough up another 120 for it!
Yeah… I’m not :P But, I am plotting a DIY solution. A solution that will probably cost more than $120 on components but, I think it will still be worth it.
So, I’ve got one for my steam deck and it’s less an issue than you might think, in my opinion.
When you’re focused on the screen, it doesn’t create too much incongruity when the background shifts, and it’s easy to just let you brain parse the screen as something that just floats in front of you.
It’s not immersive enough to get the inner ear involved and confused. It’s a lot closer to holding a phone sideways about six inches from your face and moving your head around.
The only time it felt weird was when I was using it in a well lit room, and I shifted my focus to something not on the screen, that was closer than the apparent distance to the floating display. It was weird feeling my vision try to reconcile that the nearer thing was moving behind the far thing.
Oh, you mean the problem they refused to acknowledge that was very common for a number of years? I wish I could get a refund or replacement on my 3 OFFICIAL NINTENDO controllers that suffer from drift. But alas, I don’t live where they do business “legally”
Joycon drift, and all other thumbstick drift, is already a solved problem.
Use bushings that actually have some abrasive resistance and aren’t softer than a fingernail.
Use a non-contact based sensor to determine the XY position of the stick. Hall effect, optical, strain gauge, whatever, we’ve had the tech for 50 years.
The reason why they haven’t done this is one very simple reason: $$$
N64 does use optical sensors, the n64 stick is actually super precise and doesn’t suffer from drift. The n64 is a goofy controller but it is simply a great and accurate input device, and a lot of the games were really designed with that stick and notches in mind.
But it is made of all plastic and features plastic on plastic moving parts, without lubrication, so it suffers from wear of the plastic. Worn n64 sticks will actually be filled with plastic dust from the stick and gears literally sanding themselves down. The only problem with the controller is the premature wear of the stick.
It’s crazy to me that no company ever made a decent 3rd party N64 controller. The 3rd party ones were all as ridiculous as the defaults. Great console that I loved, but would have gotten a lot more out of with better controllers.
there was a hori n64 controller that looked like a normal double handle controller and it was really good, but it’s crazy expensive these days on ebay. I’ve also heard good things about the new brawl64.
The Hori Pad Mini? I had never seen that before, leagues above anything I remember being available at the time. The other looks amazing, definitely a modern controller that I would have killed for back then.
Yup. If they’d just made the bowl out of something OTHER than ABS, they would have been good. Delrin, PTFE, even a thin layer of brass or broze, and those controllers wouldn’t have had anywhere near the amount of issues they’re known for having.
There are third-party manufacturers who sell replacement bowls and sticks, made from everything from POM to steel.
the reason the n64 sticks suck is down to the stick tension construction and not really the sensing mechanism. Pretty much the thumbstick was pressed against a plastic bowl that wore away into white dust through use, making it floppy. it didn’t really have anything to do with the fact that it was an optical stick
The sensors on the N64 are basically the same kind you’d find in a mouse wheel. They work fine.
The crap part is the physical construction. There’s a lot of parts that wear down with use and cause the joystick to become loose due to the plastics wearing away.
I hear you can get a replacement digital stick on amazon for the existing switch consoles. If the white ones that came with my oled switch ever get that drifting problem, I’m gonna buy one of those replacement stick components and send it and the drifting joycon to a tech repair shop I know of (that guy might have repaired like 200 of those already, pretty much everyone in my area these days owns a Switch). I don’t wanna risk shorting out a $70 controller doing it by myself
I installed these on my wife’s joycons because she mashes the sticks like Gail the Snail. So far no issues with drift! The kit comes with all the tools you need to do the swap so it’s very straightforward.
Glancing over the patent, I don’t think Hall effect sensors are used here. Note especially the use of a fluid, and the presence of variable resistors (parts that can wear out in current-gen controllers) for each axis.
Instead, this looks like an analog stick force-feedback mechanism that could also be used for automatic re-centering:
Accordingly, in the first example, control of current to be applied to the MRF is performed in the way as described below, thus achieving both presentation of a feeling using the MRF and an initial position restoration operation.
This wouldn’t keep the potentiometers from wearing out, but with the right software, I imagine it could automatically adjust the sticks to compensate for mild drift. (I don’t know if this would work any better than plain old calibration; it’s definitely more complicated.) Also, games could dynamically adjust stick resistance, like the DualSense can adjust trigger resistance, for interactivity/immersion.
I wonder how much this would affect battery life, how long the fluid mechanism would last with normal wear and tear, and how environmentally toxic it will be when it eventually becomes e-waste.
The PDF linked in the article seems to be a scanned image, so control+F doesn’t work, but the text is searchable here: patents.justia.com/patent/20230280850
Whoa, that force feedback mechanism sounds really cool! It might not be able to force movement but it would provide resistance to movement. I could imagine it as if your player character is walking into a wall and the joystick wont let you push forward.
Seriously…idk why they are so…obtuse to fixing shiz sometimes. Granted we are talking about the same company that won’t embrace fan work the same as Sega so they’re kinda backwards imo. Esp since that stuff usually isn’t making money or is free promo for the real shiz Ala streaming and reviews. They’re not very smart imo on a lot of things and seem to punish fans for having fun with shiz too much…
I mean I doubt it. Hall effects have been on the market for ages(notably the dreamcast as a few other comments reminded me). They can’t possibly stop hall effects and mods that allow them at this point XD
You can patent a specific implementation of a technology, but not usually the principles behind that tech. Nintendo had patented this, too, but that likely has little to no effect on other hall effect joystick manufacturers.
I upgraded my joycons to the guilykit joysticks and they are wonderful. And its not even that difficult. Took me about 30 minutes for both. But it’s a shame that I had to change them at all.
None of my PS4, 5 or Switch controllers have had any drift. I even used the Joycons in Ringfit for ages, and I was sure that spending months being strapped to my leg would bugger it up.
I’m not sure if I’m the luckiest person on Earth, I just don’t use them enough, or others are doing something I’m not (smoking or vaping are possibilities here, along with greasy food fingers).
dexerto.com
Aktywne