Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
The Age of Virtual Reality is Here
If you’ve been focused on the ongoing tragedy that is the American political system, you may not have realized that we are about to experience a new technological revolution, one that has the possibility of changing the way we interact with each other, share information, learn, and play.
Next week, hundreds of thousands of people will begin receiving their consumer versions of the Oculus Rift and HTC Vive, two competing systems for transporting people into a computer-generated reality. These systems consist of headsets with high-resolution, wide field of view 3D displays, coupled with tracking systems to measure your head and body movements. The HTC Vive also comes with hand controllers that can be tracked with sub-millimeter accuracy, and room sensors that will track your entire body, also with sub-millimeter accuracy.
The holy grail of VR is to instil a sense of “presence” in the user; i.e., to trick your brain and senses into believing that you are actually in the world projected in front of your eyes, and not just watching a 3D screen. Achieving presence requires a lot of tricks: The displays must respond instantly, with extremely low “latency” and high refresh rates; The resolution must be high enough for your eyes to perceive the scene as real; And the field of view wide enough to encompass your peripheral vision. Body and head tracking has to be accurate below a millimeter. Sounds have to be positional, and change as you move your head. All of these techniques trick your brain into accepting the virtual reality as though it were real.
This level of presence only became technically feasible in the last couple of years. Until now, virtual reality was a nauseating, pixelated potential technology waiting for the requisite advances in sensor, display, and computing technology. That time is now here. I have used this hardware and, believe, me it delivers.
It’s hard to explain what this is like. There seems to be two types of people: Those who have not tried VR and roll their eyes at the hype. They will point out that 3D TV flopped, and this is no different. They believe VR is a passing fad, or at best a niche product.
Then there are the people who have actually tried it, and almost to a person they become instant converts. It’s very hard to describe the first experience of being in a VR world with one of these devices. If you are familiar with the holodeck from Star Trek, that’s probably the closet analogue I can think of. You honestly believe that you are standing in the world the computer has generated for you. It’s unbelievably cool.
Here is an example: This is a video shows Sony’s The Deep demonstration for its Morpheus headset. In this demo, you are teleported into a shark cage and descend into the depths:
Remember, if you are wearing the headset you’re not just seeing this as a video, or even a surround video – it’s all happening in 3D space, and you feel like you are there. Everything has the correct sense of depth.
The applications for virtual reality are unfathomable. Obviously, in the beginning we will be playing a lot of games. Imagine a flight simulator where you are in the cockpit of a P-51 Mustang – look around, and it’s the real thing. With a proper joystick and throttle setup, it will even feel real. A car racing game with a force feedback wheel will be the closest you’ll ever get to actually racing in a real car without being there. Role playing games, first person shooters, and other games will be transformed.
Here’s another good video that shows several VR demos: standing on a sunken ship, climbing Mount Everest, Flying a space fighter , and Portal VR, which teleports you into the universe of the Portal game. (It also contains some zombie carnage starting around the three-minute mark.)
By the way, the accuracy of the controllers is so good that shooting a virtual gun feels like shooting a real one: you aim with your eyes through the sights, and if your aim is true, you’ll hit the target. It’s amazing. Archery games feel like shooting a real bow, and you aim the same way you would in real life.
But games are just the beginning. VR gives us the ability to share experiences the way we share photos today. Manufacturers are now building cameras that shoot entire 360 degree spheres of image. That 360 degree image can then be used to create whatever view you should be seeing when you move your head around. Imagine a camera like this recording the superbowl from the 50 yard line. When you watch that video, your view isn’t locked to the camera as it is today – you can swivel your head around and look anywhere you’d like, just as if you were there. You will feel like you are sitting on the 50 yard line.
These cameras are becoming available as “action cams.” The International Space Station has a 360 degree camera mounted on it. Viewing that video stream, you will feel like you are sitting in space looking around – in near real time. Imagine new landers going to other worlds with such cameras on them. In the future, instead of seeing photos from Mars, you’ll be able to transport yourself to Mars and just look around at whatever you would like.
On a more mundane level, imagine shooting family vacation video using a 360 camera, and then years later being able to re-live the experience as if you were there – even to the point where you can look around and see things you never saw the first time. Sharing a vacation experience with friends will let them see it just as you did – as if they were there with you at the time.
Training and education may be revolutionized. Imagine learning archaeology by being able to virtually transport yourself to real dig sites and walk around them. Imagine being able to learn auto mechanics by having a 3D model of an auto engine right in front of you, with virtual tools to disassemble it, and exploded views that show you the parts. Just “start” the virtual engine, and you’ll be able to see how it all works.
Here is a video of a game where you carry out surgery on a space alien, but it give you an idea of how we can represent and interact with 3D objects in a training environment. It also shows the sub-millimeter accuracy of the controllers that come with the HTC Vive:
Telecommuting just got a whole lot more useful. Today’s corporate telecommuting systems sometimes have cameras that can track to look at the speaker in a room, and there may be large-screen TVs connecting distant locations. Or, we simply connect using Webex or some other tool that creates audio conferences and exposes computer desktops for sharing slides and such. With VR, you can telecommute into a virtual room where multiple people can be sharing information on different whiteboards, where side conversations can take place just by “walking” over to someone, where quick notes can be jotted on boards that everyone can see, and in general where almost all the advantages of being in person are intact.
Virtual Reality is going to be great for the poor. Want to see the Louvre? Or climb Mt. Everest? Or walk along the top of the Great Wall of China? Experiences that were only available to the wealthy will be available for everyone, and for cheap. Travel costs will decline. The need for living and working space may be reduced if people spend more of their time in VR.
These are just some of the obvious things that will come from VR. The truly exciting stuff will only be known once these devices land in the hands of millions and the engine of capitalism and competition causes the kind of advancement we’ve seen with personal computers and smart phones. We don’t know what the future is bringing, but it’s going to be fantastic.
If you want to get started in VR, it’s still fairly pricey. You need a computer powerful enough to render the graphics, which will set you back about $1,000 if you don’t already have one (no Macs have this ability, so it’s a PC-0nly thing right now), or about $350 for a graphics upgrade if you have a powerful PC but not a powerful graphics card. The headsets themselves run from $599 to $799.
There are less expensive options if you want to get your feet wet. If you have a Samsung smartphone, you can buy a Samsung ‘Gear VR’ – a headset that allows you to slide your phone inside to use as the display. It’s only about $100, and doesn’t require a PC at all. Or, if you have a PlayStation 4, Sony has announced a VR headset for it, so no PC required. That will ship in the fall. You can even get a $10 cardboard headset from Google that holds your phone and gives you a similar experience as the Gear VR. These cheaper options have lower resolution and lack some features such as body movement tracking, and the sense of presence may be fleeting, but they’ll let you get familiar with the technology and they are great platforms for watching 360 degree videos and other simple VR experiences.
I’m traditionally a skeptic of sweeping claims made for new technologies. I never thought we’d see widespread flying cars. I knew the Segway wasn’t going to transform our cities. I think high speed rail is a boondoggle in most cases. Attempting to switch our energy infrastructure to renewables is a pipe dream in the short term.
But VR is the real deal. It’s here, and it’s going to be transformative. It might take an additional hardware generation or two before all the features are in place and the cost comes down to where the mass market can afford it, but it holds the potential for changing the way we live at least as much as the smart phone revolution did. If you get the chance, you really need to try it out. Until then, you won’t know what you’re missing.
Published in Science & Technology
Notice that in the shark demo, the “player” doesn’t jump back in fear when the shark attacks the cage. Why? Because the immersion is not total. He does not feel his body being shaken. He doesn’t smell the SCUBA mask. He can feel the cords and game controller.
Though there will continue to be technologically improvements in this regard and many developers would love to deepen the immersion, this bit of emotional distance might ultimately be a boon to the VR industry… especially in early iterations, as consumers warm to the concept.
Eventually, VR developers might need some sort of kill switch to prevent users from being overwhelmed in harmful ways. During the panel discussion I linked to, one person recalls a rockclimbing game in which he couldn’t force himself to take the first step off solid (virtual) ground. What would have been his body’s reaction to falling off the side of the mountain and hitting rocks in the simulation? Extreme fear can harm the body or cause a person to panic, perhaps fleeing while forgetting the chords tethering oneself and thereby injuring one’s neck.
Undoubtedly, VR devs are already taking measures to soften the blows. For example, if one falls, the simulation might fade into a less realistic illusion, reminding the player that it is just a game. But there will be mistakes and oversights.
Augmented reality is another emerging simulation technology that could revolutionize the world.
The nausea problem has been largely solved. A lot of research has gone into ways to prevent nausea, and what was discovered was that it’s a combination of proper game design, a much higher refresh rate and full head and body tracking.
The HTC Vive is not subject to nausea at all in most games, for the simple reason that you now have full room tracking and full body tracking, so what you see with your eyes does always match what your semicircular canals are telling your brain.
Of course, it’s always possible to design an experience that will make you feel sick. A virtual roller coaster will make you just as ill as a real one. But we now know enough about how this all works that there are good software development guidelines which, if followed, will eliminate nausea.
I have an Oculus DK1 – the original developer kit. It makes me very sick. I can’t use it for more than 5-10 minutes before I have to go lie down and wait for the nausea to pass. Low frame rates and no positional tracking were the culprit there.
The newest generations of hardware have none of this. Palmer Lucky himself is extremely prone to motion sickness, and he was the guinea pig for testing the hardware and software. Oculus refused to sell their headset until Palmer could use it for an indefinite amount of time without feeling nauseated. Mission accomplished.
Augmented reality has amazing possibilities. Not just for games, but for overlaying the real world with useful information. Imagine an auto mechanic running a diagnostic, then simply looking at the engine through AR glasses and having the trouble spots color coded for him. Or an instruction ‘manual’ that overlays the next step right on the device you are working on, or puts virtual arrows in space pointing to the controls you need to adjust.
AR isn’t an alternative to VR – it’s a separate technology with its own advantages and disadvantages.
I think you have to get to your next para to see why:
No, the howl of existential misery is the problem.
I am one of the people who marvel at the flexibility of the pronoun ‘you’. Today, I am amazed by your use of it. It usually conceals more than it reveals, but now it’s doing heavy duty. Who is ‘you’? Me? Certainly not! Have my enemies been slandering me! Is it ‘anyone’? That’s pretty wishful thinking…
As for the rest, you might as well say: Finally, prison will be freedom! I do not believe this to be true–I think you make possibility into the most deceptious abstraction!
I expressed similar thoughts in a post last year.
Triple-A video games are already moving in the direction of “open worlds” that simulate large and complex environments with various opportunities for interaction. And many of those attempt to recreate real locations and real historical events.
The first Assassin’s Creed game relied on historical records to simulate the ancient cities of Jerusalem and Acre. Subsequent AC games have been set in Renaissance Venice and Florence, revolutionary Paris, industrial London, and the pirate-infested Caribbean waters. Mafia 3 will recreate 1969 New Orleans. Tom Clancy’s Ghost Recon: Wildlands will be set in modern Bolivia. The level of detail and freedom in simulations today is stunning. It improves with each passing year.
It will be a while before we see large and open environments in VR. But the virtual worlds are coming.
Speaking of full room tracking, might there be a problem with necessary space?
Microsoft’s full-body motion control tech failed to catch on with consumers in part because it required a large and clear room as its setting. Many consumers did not have 5+ feet of clear space in front of their televisions and no nearby objects to accidentally knock over.
Is VR something one can use with limited space or while sitting on one’s couch? Or does it require rearrangement of furniture and similar concessions?
You’re talking about ‘presence’ – tricking your brain into believing that you are really in the space represented in the computer. That’s the moment when your perception shifts from thinking, ‘Wow, I’m looking at a very real looking 3D simulation,” to “How did I get into this strange world?”
The brain’s a tricky thing. It doesn’t need to have every sense provided for. It’s perfectly capable of filling in the gaps, so long as some critical thresholds are met. And I have seen plenty of videos of people jumping back or falling over when they were scared by something in VR.
There was an interesting demo a while ago of a plank stretched across a chasm. All you had to do was walk across it. Of course, in real life you’re just standing on a carpeted floor, but watching people try to do it wearing VR gear, it’s clear that they feel all the vertigo and fear of the real thing. They struggle to hold their balance, and even fall over as they would on a ‘real’ plank. So it’s pretty darned immersive.
Of course you’re right that the brain knows deep down that it’s not really in danger, so you’re not going to build panic-inducing levels of fear – maybe. There are a couple of horror games out there that are supposed to be truly terrifying in VR.
Vive’s implementation is pretty smart – you set up your room scale to match how much physical space you actually have, and the game adjusts to that. So while the system can provide a 15ft X 15ft room coverage, giving you a nice big space to walk around in in VR, it can be used in as little as 3.5ft x 3.5ft, if I recall correctly. And of course, if you don’t have any space at all or you are confined to a bed or a wheelchair, most games and experiences can be done as a sit-down experience as well. The Oculus Rift doesn’t have room scale at all – you can either sit, or you can stand in place and move just a little. So there are plenty of options for people challenged for space.
But if this becomes as important as I think it will, you’ll see people converting rec spaces or spare bedrooms into VR rooms. Most people can figure out how to clear a 5 X 5 area, even in a small apartment.
O to have an open-ended prelapsarian Rapture or Columbia from the “Bioshock” games.
Absolutely. And I wouldn’t be surprised if much of the ‘slack demand’ we are seeing in the economy is coming from the fact that people spend so much time online or watching big screen TV’s that they really aren’t as motivated to buy the amount of goods that their parents needed to live the ‘good life’. These days, millennials seem to be much happier living in a small apartment and riding public transit. But they’re miserable if they don’t have a good high speed internet connection and a computer.
There’s a recent trend towards ‘micro homes’ that is really catching on in a lot of places. These are homes smaller than even a house trailer – a few hundred square feet. If you’re spending most of your free time sitting at a desk on the internet, what more do you need?
Virtual reality is going to accelerate these kinds of trends. That will take pressure off the infrastructure, reduce our energy requirements, and lower the impact of wealth inequality. All voluntarily, without the government having to step in and mess it all up.
And just wait until the online economy explodes. Not just ordering from amazon, but new careers like virtual architect or designer – people you might hire to build you a customized online virtual living space, or some incredible virtual clothes. There are already people doing this for a living in virtual realities like second life. The new ‘music video’ industry might be composed of designers who build unique experiences in VR – not necessarily just depictions of the real world, but entirely new ways to engage you with combinations of sights and sounds.
We don’t know where this is all going to go yet, but it’s going to be very interesting and fun.
I’d like a Skyrim that lets me hide in bushes or peek over a boulder with my bow-and-arrow.
There will be more opportunities for open world VR when they eliminate the headset cords. It will be interesting to see how movement inputs change as well.
And just wait until they combine this tech with improved voice control tech. We might be having conversations with virtual characters by speaking aloud soon.
There is already a VR port of ‘Skyrim’, and I believe of Fallout 4. So those open worlds are already here, and they are only going to get better.
The entertainment aspect is always fun to discuss, but the industrial applications are the ones that fascinate me. I mentioned BIM above. One of the troubles we have now is that no matter how much we work in a 3D model on screen, it’s still a 2D environment and we’re prone to 2D thinking. The technical side of building engineering can be especially prone because we see the plans as finished product.
Virtual reality will help break out of this 2D thinking.
I thought of a good analogy for VR – especially the 360 video application:
It’s a ‘reverse transporter’. In Star Trek, if you wanted to visit other places they’d use the transporter to convert your body into electromagnetic radiation, then beam it to the new location.
With VR and remote cameras, We take the remote scene, convert it to a digital data stream, and teleport it into your brain. The goggles are just the delivery mechanism. So where the Star Trek transporter beams you to where the action is, this takes the action and beams it back to where you are. The end result is similar – you get to experience a remote event as if you were there.
However, VR is also a time machine. Imagine as we collect millions of 360 videos of our era. 100 years from now, people in the future will be able to walk the streets of 2018 and relive the experience as if they were there at the time. Famous events like presidential addresses, space shots, wars, and sporting events will be recorded with full 360 degree information, allowing historians or any interested people to relive them whenever they want – to the point of being able to look around and watch how the audience reacts, or look up to see what planes were flying overhead, or whatever. You’re no longer locked to the visual scene the videographer thought you’d want to see.
If you have small children today, imagine how cool it will be to be able to go back and relive some of those moments when your kids are grown and moved away. Sure, we do that with home movies today, but imagine being able to put yourself right back in the head of the ‘you’ from 30 years ago, and see things again exactly as you saw them then.
Absolutely. Architecture will be revolutionized. Imagine designing the building in CAD, and then being able to walk through it virtually and see every space, every light, every shadow. Customers will be able to get exactly what they want.
Thinking of redecorating? Have your room scanned and digitized, then step into it and change the colors, change the furniture and curtains, and play around until you’ve designed a space exactly the way you want it. Then just press the ‘order’ button, and all the required goods are added to a shopping cart. I’m sure online vendors are working on this kind of application right now.
How about art? I want to see the world’s greatest sculptures in 3D. I want to walk around them, be able to lean in and examine details, look under them, etc. That’s coming very soon.
I totally agree with this. We will soon be able to watch movies from inside the movie. Perhaps we’ll even be able to re-watch by picking a different character’s point of view and see the whole thing again through their eyes.
Then there’s interactive storytelling in a free-form universe, re-enactments of famous battles with you as part of it – imagine VR Call of Duty with story elements written by top writers and the world designed by people who will specialize in this kind of experience. It’s going to be awesome.
That’s going to be tough. For VR to work, the latency between your head and body movements and what you perceive has to be incredibly small. So much so that the newest video cards from AMD and Nvidia have special modes where the frame is rendered based on what you are looking at, and then before it’s sent down the pipe a second query is made to see if your head has moved during the time that the frame buffer was being created. If it was, the final image is warped to match your new head position before sent to you. Even that tiny delay required to build the image is enough to cause issues.
We currently don’t have a wireless protocol with latency even remotely good enough to transmit video that will work in VR. Just the time required to encode the data, transmit it, and decode it is currently far too long.
I think it’s more likely that we’ll be using wearable computers, rather than wireless video. But I could easily be wrong – VR is going to encourage investment in all kinds of R&D around this, so who knows?
This is really exciting. There is also a lot of work being done on facial warping techniques that allow virtual characters to ‘speak’ in real time, with their mouth movements exactly matching their words, and their facial expressions exactly matching yours. Very soon you’ll see applications where the room scanners and cameras that are part of VR will map your body and face and put a virtual avatar of ‘you’ into the VR world. A Ricochet virtual meetup could be done with life-like avatars of each person, interacting just as they would in real life. That technology is pretty much here now, just waiting to be mainstreamed.
Check out this video, where youtube videos are actually manipulated to overlay your own expressions on another character:
High fidelity facial capture:
The latest Hitman game provides a similar experience. It places the player in a large and complex environment full of scripted characters and events. As the player moves through the environment, those characters and events are following their scripts even when the player is not within view. As the player acts — with countless options for geographic approach, strategic order, manners, methods, and accidental encounters — the NPCs (non-player characters) respond.
So the player can explore the same world with the same characters doing the same things (until acted upon) repeatedly and yet enjoy a different experience each time.
Now imagine this general design concept being applied to a less nefarious genre of interaction. For example, you could mingle among a gathering of famous philosophers (“Plato, my friend! Meet Descartes.”) or SCUBA around the Great Barrier Reef. Watch as a 16th-century cathedral or modern factory is constructed and see what each craftsman is doing.
The point is: game designers are already practiced in some of the necessary preparations.
Lovely. But isn’t Mars an appealing destination, in part, because it is (for now, at least) inaccessible? If my neighbor’s house — not a particularly interesting thing in itself — were floating about in space, 140 million miles from my body, I might spend vast sums of money organizing an expedition to document it. Why? Because it would be an unknown.
Transportation, photography, and the Internet may have pruned the universe’s enigma, but virtual reality will take a chainsaw to its trunk. (And we ought, for the free market’s sake, to permit it to do so.)
That might be way too cool. I finally just beat Infinite last night. If you put aside the somewhat derivative style of character revelation — which was extremely well done regardless — it may have been the best in the series.
You bet! And … getta loada the date!
How are the advertisers gonna screw this up? Google ads in the VR? All the stuff I have searched for recently shows up in my VR with little price tags hovering above?
Now there’s an idea – a VR presence at high-profile events. Like, say, paying five bucks for a VR seat at the Superbowl. (And maybe $50 for a skybox filled with celebrities and supermodels.) A good promoter could make a killing off that. A killer app, so to speak.
I’d also be interested in a VR seat at various events and even public places; like a webcam except you’re in the environment, rather than watching through a window.
It is remarkable, isn’t it? Play the expansion packs – one of them takes place in Rapture before the fall, in all its shiny Moderne Randian glory.
I got a Google Cardboard and it didn’t work with my Blackberry, therefore I say Virtual Reality is bunk.
I am confidence there are no holes in my reasoning.
Canadians seesh.
Planning to.
They have force feedback vests available. This will fix part of the immersion.
Next you would need something for your feet, like a treadmill.
HTC and Valve put out a helpful advertisement for the Vive, using a green screen:
Is local processing the only hurdle that must be surpassed to cut the cord? If that can’t current be added to the headset without making it uncomfortable, could it be accomplished via a backpack PC?
The problem with wireless is latency. The time required to encode the ames, transmit them, receive them and decode them just adds too much lag, and VR absolutely depends on imperceptible latency. A backpack PC would work, though. The necessar graphics card would eat batteries like craz, though. So it would have to be a heav thing. Give it a few years.
I received a Vive today, and set it up. I’m currently going through a VR re-enactment of the Apollo 11 mission. It’s amazing.
I watched an interview with a guy who spent two hours in a VR simulation (and many more hours split between a variety of games he was tasked with reviewing). He said returning to reality after that much time “in the box” resulted in a strange feeling, like psychological lag. It quickly passed.
My old philosophy professor suggested that the way we know reality from dreams is continuity. Reality always picks up where it left off. If VR survives long enough to be worked into large continous environments like Skyrim, it will be interesting to see what psychological effects that might have on some users.