A Virtually Realistic TechStuff

improvisational performer with moments of genius. And Rubens also written for video game outlets that we often have conversations online just about like issues in the video game world. This is absolutely true. We'll just so there and say like, oh man, it's crazy in it, and then uh, I'm like, yeah, I can't believe that Polygon gave that a six point five when clearly was a seven point four at least,

what's that about it? Together game journalists? But I let I wanted Reuben to come onto the show, and I gave him, of course, the opportunity to choose whatever topic he wanted to talk about, and uh, you really want to talk about virtual reality? Yeah? To escape my own perilous and terrible life, I can totally understand that. I mean, we has been the promise of virtual reality ever since we first heard the term. Do you remember, you know,

approximately when you first heard about virtual reality? Geez um, So I guess for reference, like I'm twenty eight, so one of the first times I can I don't know, probably like lawnmower Man or Johnny Demonic is one of the first pop culture references I really saw to it. Yeah, lawnmower Man was like, that was like the thing that totally destroyed virtual reality in a way because the well one Klant's lawnmar Man. It's terrible movie, but too well hey, no,

completely right now, It's awesome in its own way. It's just not well made. It's still entertaining, just not the way it was necessarily in ended to be in relative to Johnny Nemonic, I mean that's true. We're we're talking, you know, in comparison, but iced tea. Yeah, I mean, that's that's a tough competition any way you slice it. But the issue, and we'll talk about this more in a little bit, was that the reality of what virtual reality was was capable of doing versus what it was

being depicted as doing was totally different. Um personally. I remember hearing about virtual reality in the early nineties. I grew up in seventies and eighties, and but I remember by the early nineties of a teenager and even one of the malls here in the Metro Atlanta area had a small virtual reality game center where you would go in, stand on a platform, have a fifty pound head mounted

display put down on you. It was actually suspended by cables because it was too heavy to put on just kids, and then you would uh make yourself film nauseated as you laid against polygone pterodactyls that flew and looked nothing like anything that ever really existed. Ever dinosaur adventure, but every dinosaur is just a couple of triangles. Yeah, it was pretty much. That was it. And then you had a little a little kind of prism looking like gun

that fired perfect cube bullets. Yeah it was. It was pretty limited, but it showed what the promise of virtual virtual reality could be. And I think a lot of people focused more on the limitations than on the the actual potential for the technology, and that really set the entire industry back. But just as a quick overview of the actual history, not just our personal history, you know,

the basic concept is pretty simple. It's the idea of creating a virtual environment that mimics a physical environment, so that the person who is in whatever set up you're using feels as if they're in a real place, but it's all virtual representation. So uh, two very important concepts with this are immersion, which is your sense of being in that space, and interactivity, which is your ability to

manipulate things within that space. So if you don't feel immersion, then there's no illusion that you are in this environment and it all kind of falls apart. And if you don't have interaction, then it's like you are in a an immersive movie, but you have no agency, You have no ability to affect the things around you. Even more than than it's sort of disconnecting when when you don't have that one to one ratio of your own movement and immersion happening, it can be complete disorienting. Oh yeah.

The big issue of VR is when you have that lag, it can be what causes when people get really sick from it. That's a major issue that happens. Oh sure, Yeah. Latency is a terrible problem. I've I've had that experience as well. In fact, I had it with that pterodactyl based game, and that was super simple graphics, so there's very low latency. There was still enough where it doesn't

take much for humans to pick up on it. We're talking on the millisecond frame right, like something like sixty milliseconds. Anything more than that you can start to kind of feel it. It's one of the big challenges of the Oculus Rift was to bring that latency number down as far as possible. But then, you know, I've even used the Oculus Rift and while the latency wasn't so bad, there was a disconnect in the interface that made me feel a little swimmy. UM. And I'll talk about that

a little bit later too. So we've got some interesting examples of precursors to what we consider the typical virtual reality gear. And I love that you pulled these up in your research. Uh, going all the way back to the nineteenth century here. Yeah, we're talking. UM eight. The wheat Stone stereoscope. A stereoscope technology is a major major component to virtual reality. Um. It's still being used, I believe in Occulus Rift the current iterations to make that

three D immersive experience. Uh. And so we had these Uh, like the wheat Stone stereoscope. It was basically a pair of giant or is at forty five degree angles that would make a stereoscopic image. Um. And we have this sort of going on until we got the home Stereoscope in one, which is a handheld version. Uh. And it looks you can see photos of online. It looks very, very similar to sort of a steampunk wooden version of a virtual boy. Um. It's just it's a little mobile, handheld,

handheld put it on your face sort of interface. Except I would imagine I don't know, maybe the headache is quite as bad as virtual I sure hope not. Man. Did you ever play with one of those? Yeah? And it was heinous, right, Although it did introduce the world to Warrior, which is great. I mean Warrio Mario Tennis I think was on there. But it also made your eyes bleed. Yeah. No, not not the most pleasant of

user experiences. Yeah. And stereoscopic effect is is pretty cool because there's a technological element, but the really important step is all done by your brain, because it's it's your eyes getting too slightly different images and then your brain combining that to create the what you perceive as a three dimensional base. It's the same sort of principle that comes out of three D movies, the idea that you're you're getting just slightly different views to kind of mimic

what you would get with parallax. That's where your eyes converge on a point and you get that sense of depth. So super interesting. And then nine six you've got the tell usphere mask. Yeah, this guy more Highlig he created this. It's a stereoscopic three D TV with stereoston and wide vision display and it's probably the first thing that I've found that's similar to the head mounted displays we're kind of more used to in looking at Project Morpheus or sorry,

Project Morpheus rather and Oculus rift um. And then uh, two years later, the same guy makes this thing, the Sense of Rama, which is an Arcad style three D cabinet with a vibrating seat and scent producer um. It reminds me of a lot of there is Disney rides. That there's an alien Disney ride actually that you would sit in this theater and there was three D, but they would like you have water splashed on you when the alien are crawling around a little like rubber tubes.

It hit your legs. It's that sort of full immersion, same sort of thing with they have the It's Tough to Be a Bug movie that's in the Animal Kingdom, Disney Animal Kingdom, same sort of stuff where it comes complete with the three dimensional movie. You get the sense like when the stink bug does his thing, you get a nice waft of a suitably stinky but disneyfied stinky smell. Yeah, you're hope, you're hopeful that you know that that truro

that your your kid ate is not what's causing the problem. Uh. And they even had um things under like incorporate in the benches you sit in so that when the wasps attack, a little thing pokes you in the back, or at the very very end, when all the lights have gone down and you haven't been they haven't come up, so you can get up and walk away. Yet Uh, little things directly under the seat, like under your butt move. So it's supposed to be all the cockroaches. Yeah, it's

little creepy, but you know it is very immersive, terrifyingly immersive. Right. And then we have Ivan Sutherland who created the sketch pad. So yeah, yeah, I like this. Note you have to about the precursor to CAD programs, computer assisted design programs, and our computer aided design depends on who's spelling out the acronym. But you know, the graphic user interface or

derived from this as well. Uh, this is one of those things that ended up making a huge impact on computing in general, not just virtual reality, but really the way that we interact with our devices today. And it's funny because this was in the VR headset he made, which they called the sort of dominicles because it hung

from the scene. Of course, because it was so heavy, and of course it's a nerdy name as long as long as you didn't undo the Gordian knot totally okay, But it's funny because this is and yet this sounds very very similar to your Pterodactyl adventure set up. Yeah. Yeah, so the technology it was massive and heavier for a while. Yeah. I think a lot of these displays too, especially the early ones you're talking about, probably cathode ray tube technology

and a cathode ray tube television. It's enormous. I had to have a space for a vacuum tube in it. That's the tube part to create the energy for the electron gun to fire the electrons that create the images. So yeah, you definitely needed to have some assistance there to to carry that kind of weight. I mean, you know, I don't I don't know how familiar you are with

CRT displays. Yeah, well enough. Yeah, if you've ever had to move one and you think, yes, I want to strap this to my face, probably not one of the first things that occurs to you. I don't have you seen my neck? I know, it's kind of like I was thinking that, you know that you were just maybe a little angry and went a little hulk ish, but just in the neck area. The only thing I work out of the gym is my neck. That's good. Well,

you know, it's it's important to concentrate on muscle groups. Yeah, anxactly. And then when we get up to nineteen eighties seven, that's when a guy named Jarn Lanier created, or at least as credited for creating, the term virtual reality. There's some dispute over whether or not he actually coined it, but he certainly popularized it. Now he was a computer scientist, still is a computer scientist, I should say, because he's

still very much active. But he worked for Atari for a while and then Atari, I don't know if you know this, Struben, but in n at kind of blew up. I've I've I've read. Yeah, I was, I was not alive. I was there, Reuben, I was there. I've seen things, Ruben. You can't unsee's eye. When Atari died, uh, you know it, Actually it was. It was a restructuring and Lanier was let go and he co founded a company called VPL Research, and then uh he worked there for quite some time.

He's also done work for Advanced Networking Services as well as for Microsoft Research, and he's been involved in tons of big technology discussions, not just virtual reality but web two point oh, which I think he argued didn't really mean anything. I think I would agree with him on that. I think it was more of a marketing term than anything technologically. And then Uh, he's also weighed in on

discussions about singularity and futurism. If you think of Ray kurtzwile as we're all going to be immortal robots, uh and they're going to take over. He's kind of on the opposite end of the spectrum, which is interesting because I see a lot of coverage on the kurtswild side and very little on the skeptical, like, wait a minute, do you really think we're all gonna be robots? We all like a fairy tale. I think we want to that pine, this skuy dream of put my brain in

our robot body. Get that molder I want to believe, poster on our on our walls. Put my brain in a molder body. You know, I'll sign me up. That is a huge upgrade from where I am right now. But Lenar worked with Tom Zimmerman, who was the guy who invented the data glove. Uh, the data glove being the foundation for things like the Nintendo Power Glove. The data glove was better, but yeah, he designed the data glove.

And then so Laniar and Zimmerman work together to create some of the early examples of the typical VR hardware that we see today, like the like the interfaces, not just the head meld displays, which, as you have you

as you have pointed out, predate this. So Lanier and Zimmerman work together on creating a lot of the hardware that we associate with virtual reality, not just head meld displays, which obviously pre date their work, but also some of the other interfaces like how do you actually move around and interact with this virtual environment. Because it's one thing to see it, I mean a head meld display where it tracks the movement of your head so that your

perspective changes as you as you look around. That's one thing, but it's still that doesn't still give you the ability to actually do anything in that space. Certainly, Like we previously mentioned that one to one ratio, it's one thing to like to see with your head and move around, you know, with your eyes um, but to actually to try to mimic walking if it's an omnidirectional try to mill um holding whatever you're holding in your hands for this and we so we see more and more interfaces

being added on. This is something we see in the Oculus Rift is sort of just one ingredient in the full immersion that they're trying to tell you one exactly. Yeah, and that's is that's one of the biggest challenges. I think we'll talk about some of the challenges of VR, but interfaces alone, like how do you get an interface that is transparent enough, not not physically, but transparent enough where you're having a natural experience in this unnatural virtual

environment so that it fades way. The last thing you want is for the user to be hyper aware of the interface. You want that to fade out. So let's talk about some of the gears. So we've already mentioned displays. Headmond display is probably the most popular, although not the only one. There are some displays that are an entire room where things are projected onto walls or uh like a sphere where it's projected on the inside of the

sphere and you sit down in the middle. So that like like a lot of aircraft simulation is done in this sort of style. But the one that we're mostly familiar with, especially if you're talking about a consumer level, is heading out a displace because most of us can't convert room in our house to VR hardware. That's my first question for a real tear is there, Yeah, there can't be any windows, and no windows. Is gonna be fully internal room, a perfect sphere if possible? Yeah, that

would be. That would be ideal you crafted outside of zero g exactly. If it's completely out, if it's out of true even by a little bit, it's going to completely throw off my game of Skyrim. And I really I'm really close to the end. I pull my laser micrometer out to make sure this thing is perfect. Yeah, yeah, that might be a little bit um well excessive, I

think would say. Yeah. So, the latest version of the Oculus Rift developer kit, you know, because this still has of the recording of this podcast, isn't a consumer product, still in the developer phase, but the latest version has tracking dots on it which an external camera can can track, and it will be able to measure the different distances between the different points and thus be able to tell

what orientation the mask is on. So not only do you have the tracking technology built into the actual device where it's got accelerometers and that kind of thing to tell whether or not you're looking left, right, up or down,

it can actually tell the angle. So, for example, if I wanted if I'm looking at a virtual table in front of me and I want to see if someone has written something on the underside of the table, I can bend down and look up and it was able to track that change an angle, so I can see the underside of the table in the virtual world, which is pretty cool. It's another one of those ways of adding a level of immersion. Oculus has been really careful to say this may not be in the final consumer version.

You know, you might not have this external tracking element because it does require even more hardware. You have to have that webcam to all the bust computers I think are made with them now. Yeah, but it's it's one of those things where it really does add that extra element of interaction. Well, looking under tables is if you're in a Hobo simulator, you've got to be able to look under furniture to see Hobo marks to what's going on.

That's right, it's a big one one. Yeah, I can I can imagine anything, well, anything where you're having to look around a corner, like especially like survival horror games. I mean, this will be so useful if you have the guts to play a survival horror game that fully immerses you. I'm thinking of PT the playable trailer. Yeah, yeah, can you imagine if that were an innoculous riff situation you probably you would probably never stop screaming. It's it was,

it was. I'm not easily scared by by any sort of media. Usually that game was terrifying, So I can't imagine like being you can't like not being able to escape it, right, Yeah, you can't look away from the screen because the screen is directly in front of your eyes no matter where you look. Yeah, exactly, especially if you have three dimensional sound and and uh, really good

headphones as well. That's what's really going to create that that incredible experience that is probably gonna be unforgettable, and not necessarily in a good way. It's interesting. So again we're talking about this sort of this ecosystem you have to build around not just the headset, but so the the externally amounted camera. So like this Rift, they have those points on the front to kind of to inform the environment more of what angles you're looking at, etcetera.

A thing they've run into is that they don't have a way to track the back of the head. Currently, not a hard thing to fix. Um the Project Morpheus, which is Sony's thing, because they're using um light instead of these little marks, they're using led lights on the front and the back of the set. UM. It's actually a way too. If you turn around, the camera knows that you're fully turned around, not just your vision, but

I'll track the back of your head. There's there's a big deal because when I was playing with the Oculus Rift just a week ago. UM, one of the big issues I ran into is that it yeah, I could turn my head to the left, but if I wanted to actually change the orientation of the character character's body, I had to use a controller to turn the character to the left, which would then end up creating that disconnect I was talking about, because that's not the way

we normally interact with our environments, right. We don't normally hold a controller in our hands and say, well, I need to go to the left now, and then push a thumbstick over. So it's very disorienting. Something where it can detect exactly where you are, you know, whether in a full degree kind of way, UH could potentially allow you to actually change that orientation in the game without having to use a controller to do it. So if I turned to the left, my character actively turns to

the left. You have to balance that out because do you automatically make it so that the character is moving in the direction you're looking, or do you want to have it so that you can look to the left but still keep running forward for example, And that's one of the big things in VR I know for first person shooter tap games was the ability to shoot somewhere you were not looking, because up until now it's always been contained to your vision either the center of the

screen like most fps is work or I want to say like they're like the old Golden Eye actually let you free range your cursor around the screen. Um, but the ability to there's actually a great video on YouTube of guy who's using knock this rift with a couple of purples uh in Half Life two in a shooting range and he's doing all sorts of crazy stuff behind

the back and over the over the head. I saw one of those where they were using not the Oculus rift, but a similar head mounted display and a vest, and the vest was what was tracking all the body motions. So it actually had sensors that went out to the arms and too the legs. So if you squatted, the character would squat in the video game. If you put your hand behind your back, your character would put his

hand or her hand behind his or her back. And so it was one of those things that got around that by just take can a totally different approach to the hardware. Of course, then you get into the question of how expensive would it be to produce a full suit and would people really want to have something like that. You know, again, you'd be really aware of that interface, at least at first. Maybe after a while it fades away.

But it's one of those things where again you don't want the player to be too too aware of what's going on on a technical level, because then then they're thinking about that and so the experience that they're supposed to have. And I think that that's sort of the ultimate question that comes in with mass commercial acceptance of VR is you know, ideally, if you wanted to get some sort of like motion graph motion graphics set up like mo capping. Uh, you know, I would imagine something

along those lines. Some sort of set up like that would be ideal. The whole array of cameras you're covered in little balls that say exactly where you are. Um, but obviously not price effective. I mean, you could maybe have an amusement park where that's something people can go and do as part of the amount. Man, yeah, I pay money for that. Yeah, that's all it was for a little bit. Sure, I'd be jumped right on that.

I mean, I'm amazed actually that that's not something you see it's a Disney Hollywood studios where you know, that's that's what that that the impart is all about, is about the production of film and television and all the different stuff that goes into it. Um, maybe one day we'll see that. I can't imagine that. It would be

a really slow loading kind of attraction. The wait time for this is four hours and there are three people ahead of you in line, like four days based on ego to Disney the simmer for maybe maybe you can fast pass it well other things, Like we mentioned audio, that's another really big element in immersion, right if you are looking at something, if something's kind of on the right side of the peripheral of your vision, but you're hearing stuff from the left, that's really disorienting, UM, because

first you think, oh, well, whatever that is, there's it's brother is right next to me. But then you look and there's nothing there. So you really have to have very good sound design and sound orientation as well. And there are a lot of cool ways of doing that, like binaural microphones which mimic the way sound goes into the human head. Yeah, I experienced that with UM. We were talking about SMR, which you just did the podcast on. I listened to it at work and that pretty good

headphones there, and I was just like, oh, this is delightful. Yeah, yeah, having the sound of someone slowly walking around you while talking and if you close your eyes, it's like you're there. In fact, there are a lot of a s MR videos that are about things like a like a visit to the barbershop. That's a really popular one. And you know, I don't have hair anymore, but I remember when I do, and and so so I there was. There's there's a lot where it's like the sound of scissors cutting near you,

and it's amazing how immersive this is. And that's only one sense. If you close your eyes, it's as if you were there, to the point where you can almost imagine feeling it, even though there's nothing physically touching you up from your headphones. But it's amazing how convincing that experiences. So sound is really important, so you need to make sure that that is is well done, just as well done as the graphics, if not more so, I think. Um As for the interface, that's the other issue we

were talking about. How do you let the person explore that virtual environment. Is it a controller, is it a treadmill? We talked about there there are omnidirectional treadmills out there that can allow you to actually turn and run and so that way you're not limited to just forward or backward.

I've never tried one. I'm curious about things like momentum and inertia and if you if you run in one direction and then you suddenly need to turn to another one, does it keep going so that you're stumbling sideways as you're trying to move to a different direction. Or I have seen one that looks like, uh, kind of like almost like a walk. It's it's just you know what

I'm talking about right on the teflon, sir fist. Yeah, it's meant to reduce friction to a minimum so that you can just run as if you know, you can do an all outsprint on this thing. I've seen it. I don't trust myself on it. I just for that when I imagine like missing a step and then suddenly just doing a whoopsie daisy, you know, catapulting myself into the air as I fall down and then immediately slide up through the other side. But it's an interesting approach

to a difficult problem. It's certainly since the most commercially viable version of that that I've seen, um and and they they've built in a bunch of things like stabilize yourself at the waist and all this stuff. But again, like, yeah, the question is just like basic momentum, like when you're when you see a runner, like a football runner um that they called football they they're called football runners like that. I'm pretty sure there's at least seventeen football runners ut

per side. And then, given you're asking the wrong guy, I'm sorry, Well, you know sports guys, sports ball. Yeah, when you see guys in sports running, know that ability to to to duke or quick pivot like whatever you're doing. You know, if you're if you're playing Maddened twenty billion in the future and it's vr, are we gonna be able to emulate that sort of ability, that sort of athleticism or do you want to? Yeah, there's has come a point where you say, all right, well, how immersive

do I want this experience to be? Because uh, games are about escapism, And there comes a point where you turn the corner on that right where you're no longer escaping it, you are experiencing it. If you have if you have managed to replicate the experience to the point where it's indistinguishable from the real thing. Then you're really having that experience. So then you're having the question of do I want that, I mean like a war game.

If I were really like war and not not some sort of gamification version of war, that would be a pretty traumatizing and terrifying experience. Maybe we'd have less real war. Maybe we would. Maybe we would, Maybe people's opinion about the games would change. Will say, you know, I I get why these other games are fun because they remove all the truly negative things of war and they just make it a competition, like it's just a winner lose, and so that it removes the ambiguity, it removes all

the ethics. It's it's really all about what's your skill level. But if you were to add that back in through immersions, then suddenly you've got another conversation going on. I truly can't wait till I come on from work and I see my student loan bills in the mail, and I go, I can't deal with this right now, And I put on my VR headset and I play my student loan bills right and I just stress about it. In virtual have you played you know this is this is not

exactly virtual reality, but it does remind me. Have you played Papers Please? I was exactly thinking Papers play amazing game. Yeah, because it's it's it's a it's not virtual reality, but it's immersion. Like all your all you do is the exact same job of border guard would do, and you need to do it well and fast, and you have to make some pretty tough decisions based upon your own

family's situation. And there's gamification to it, certainly, but it's pretty pretty honest to what the experiences is like from what I've seen, and that's what makes it terrified and interesting. And I'm just waiting for the Occulus riff version of Papers Please. We're all gonna we're gonna start escaping two jobs, because what's gonna happen. It's really just like I just

want some other job now. Either that either that's because you don't like your job or you just want to be able to appreciate the job you have that much more. I haven't had to work for six months. It's time to put on the cubical some of your area. This report, Oh my god, this is crazy. Yeah, here's the water cooler level. I love this level. What's going on to c TV last night? This is great. So there's a show called Lost getting back into virtual reality. Another interesting

aspect is the tactile part. We talked a little bit about that with the user interface, But how do you give feedback to somebody? And in some cases there is no feedback. It's just like a video game where a video game where you're using a controller where there's no rumble pack, it's there's no physical feedback. It's just visual and audio and that kind of stuff. But a haptic

system does give you physical feedback. A simple one is like a rumble pack in a in a controller, so like you know, you know, there are a lot of games where if you're bordering on something dangerous, it starts to rumble, alerting you. It's another way of letting the player knows something's about to happen. Um. I've seen a lot of different haptic feedback systems. They're active ones. Those are ones that have motors in the respond to things

that are happening in the game. I've seen some that were like it was like a vest that would give you an impact if you got shot, and even a helmet that would have a little knocking sounds if you've got a head shot and uh, I remember seeing it at c Yes, and they said, do you want to try this on? No? Right, do you do? You want a physical reminder of how terrible you are at Halo. I'm pretty much reminded immediately upon lugging in all of those twelve exactly twelve man that someone or nine. But

then you've got passive hat haptic systems. These are actual physical objects that are in your physical environment that are mapped to the virtual world that you are in. So a simple one could be a chair that's bolted to the floor so you can't move the chair, but the chair represents an actual chair in the virtual world that if you sit in it, your virtual character sits in the virtual one. So it could be like the Game

of Thrones and that's the iron throne there. It's really just a chair that's in the middle of the room you're in. Just musical Game of Thrones. Yeah, that's what it really could be. You can just the music plays and at the end of it, a stark dies um. But there's also a strailer. There's also there's also the idea of having physical objects that you can actually pick up, move and those are mapped to virtual objects. So the idea is that the things you pick up have a

weight to them, they have a physical presence. You can actually push them and pull them and pick them up and move them around and mean, while in the virtual world, your character is picking up whatever the virtual version of that object is. So it might be something as simple as like a basketball that you pick up in the physical world, but in the virtual world, maybe it's someone's head, someone's bouncy bouncy head, a star head. Yeah, it could be a star For those of you who haven't watched

Season one, don't get too attached to Sean Bean. Um should we ever be attached to Shan I mean, I mean he's he's amazing, but you know immediately right what's going to happen. I want them to have a Groundhog Day maybe where he just dies a whole bunch redo the Edge of Tomorrow, but have Sean Bean as the main character. Exactly Why not him is that it Tom Cruise. To be fair, though Tom Cruise was very entertaining in

that movie. I gotta watch it. It's good. It's interesting because I know Microsoft Actually, um, so you mentioned like my probably early experience with haptic feedbacks like the Nintendo sixty four Rumble Pack, Star Fox sixty four whatever, the old like nineties video game barrel roll exactly to etcetera.

Um And, Uh, it's interesting. One of the things I actually like about the Xbox One as far as feedback is concerned versus the PS four um is that they actually win like above and beyond with the rumble feedback on that if you play fours on it. Uh. They put there's like the things that make the rumble happen, these little weights that they basically spin at different frequencies, different speeds, um and that kind of will pull different

directions or give you different variances in intensity. They put some of them right in the triggers, like right under the triggers. So breaking in forsa is surprisingly realistic feeling, even though it's your fingers instead of your feet. The

feedback is really interesting. Um And and to the other spectrum, there's stuff out there now, like gloves that you'll put on and they'll if you're grabbing an object that's around, they will harden or provide resistance so that you physically can't fully close your hand because there is that object in your hand virtually, and that's really really interesting. That's

really cool to me. It looks like you're wearing like a plastic skeleton hand on top of your actual hand because because they have these sort of levered arms that end in rings, the rings slip around the ends of your fingers, so when you are squeezing your hand, you're actually pulling against these rings and these and these levered arms that have usually some sort of pivot in the center.

And there's a there's actually a kickstarter for this. There's a kickstarter for a specific glove that does this kind of thing where you map it to a virtual object like a ball. And yeah, just like you said, as your hands close in on what the dimensions of the ball would be in the virtual world, those those connections stiffen so that you get that resistance and it feels

as if you're holding something. Now granted there's no weight there, but there is that resistance so that you know, you have the feeling, oh, this weightless ball is in my hand now, which is kind of neat um. And whatever fail safes are in that glove to keep it from going the opposite ring, right, it broke every finger I had I should have known not to pick up that ball. Uh, you shouldn't have gotten the deaf kid. Maybe wait for the consumer version. There are other really cool ones that

we've talked about. You talked about the Censorama, where there was a cabinet that actually also produced scent. Then we talked about the Disney the Disney Experience. There's also a Disney right called sore In that incorporates smells in that it has three different smells. As I recalled, this is trivia for you guys. And by the way, if you ever are in a Disney trivia game, they do ask this question. The three smells are the smell of the ocean,

the smell of evergreens, and the smell of oranges. Because it's sore In over California, and you go to different environments over California, so Sorein incorporates a lot of the elements we're talking about with virtual reality. It's a huge screen that that fills up your your peripheral, so you're seeing. Everything you see is related to a a fly over sequence over California. It blows wind at you, so that's sort of a haptic feedback, getting a tactile sensation Um,

so you're getting this breeze blown at you. So it's or simulates what would happen if you were hang gliding over this space. You are in a row of seats that is suspended from the ceiling and swoops around as if you are actually flying. So that's another tactile response. And you get this scent in certain scenes where they

incorporate the scent in the breeze that hits you. So you're flying over an orange grove and you smell oranges over long Beach and you smell medical marijuana, right, yeah, you fly fly over muscle Beach and you just smell like tanning oil and yeah, you know. Sure they decided not to go the full route with immersion because it is California. But no, it was really pretty, pretty interesting, pretty immersive. And I don't know of any consumer approaches

to trying this because it's problematic. How do you produce the sense? How many would you produce? How do you clean something like that so that you don't you know, like everything smells like oranges now, you know, it's it's a little tricky. So I don't expect that this will become something that we commonly see in the in the

home market. But you have to mentorize some sort of chemical synthesis down to a tiny, tiny packet of a billion mixtures for a billion different combinations and make certain right, Yeah, it would be impossible. Or you have a really good friend who just has a huge box of stuff and it's just waiting and like you say something like, wow, that really does look like pizza. Pizza, pizza. Here you go,

and then just hold it under your nose. I mean that's really the only other word version just by twenty Yankee candles and put him in front of exactly, okay, pizza. See that's I'm gonna go with. I'm gonna go with Apple Christmas. And how's that pizza, Apple Christmas Pizza. I don't know somebody's gonna do it. Yeah, I'll sign me up. Uh. Then we have, of course, the sense of taste, which I don't even want to think about. I don't I

don't need that kind of level of immersion. I don't use one game I've played where I'm like, I want to taste what's happening here? Did you ever see the British comedy series hyper Drive as Nick for lost Uh. In it, he plays the lead character who's a captain of a of a starship and it's a very kind of hopeless crew and they mess up all the time. In one of the episodes, one of the characters has a device that is a glowing ball that you put

in your mouth. It's got a cable attached to it and in a handheld um uh interface where you tell it what flavor you want it to be. So it's just supposed to be a thing that lets you taste different flavors. And of course he hands it to a friend of his who immediately starts putting in horrible flavors

as the guy's got in his mouth. And never trust friends, never, But that's one of the big reasons, Like, even if that were such a thing, I can't imagine keeping that clean or wanting to put it in my mouth or so I'm pretty sure taste is out. Um And a lot of VR gear, whether it's meant for games, because we think of it in terms of games, but VR

has a lot of applications outside of video games. But a lot of the gear that researchers and VR use is just repurposed video game gear, And the reason for that is it's expensive to develop your own technology to do this sort of stuff. And I talked to a VR specialist a few years ago. It was actually when the Nintendo we first came out, and she was so excited about the Wii because she thought the WE controller the we mote was an incredible boon for VR research

and was a new way of interfacing with VR. Same thing with the connect, huge boon to VR research. So the interesting thing to me is that virtual environments are still very much part of the research world, but they're so dependent upon companies like Oculus making stuff because without it, they just don't have the money to go out and develop the technology. You don't want to do a one off every single time. Yeah, and it's it's sort of

I think a sad reverse that. Um, the funding for the science usually isn't there and nless we find a commercially viable thing for it, and video games are money makers, it's super commercially viable. So when we see that stuff happen, it's like, great, we have the tools. Um, that's yeah, that's been There's always the always needed to be something to kind of bring new mediums, new formats for it.

Video games. You could argue that the PS three really made blue ray take off as opposed to HTDVD because the PS three had it built in and people had that, um crazy enough. Frankly, pornography is a big boon to a lot of medium formats taking off. There's there's been Google class pornography for VR type usage. That's a terrible idea. It's terrible idea. Yeah, people, you can see what's on that screen from the other side. I don't know if

you know that or not. Yeah, uh no, but at any rate, yeah, no, we we had the same discussion. I did a podcast, a couple of podcasts with i Azactar of c net, and we talked about technology has that gone obsolete? And we talked about then that no, pornography actually is an industry leader. You start to see which standards are going to be adopted because that's the one that it moves to one first and that becomes

the standard. And it's odd. It's a surprising barometer if you, I mean, if you if there's a format war coming up, you can pretty much look at the adult industry as a good indicator of what's gonna probably went out. Yeah, it's it's there's almost a full episode there but I'm not sure I'm comfortable enough doing it, but it is research hours. I'll put it in. John. Well, that's I'm

glad to know you got my back anytime. All right. Well, you know you've got some interesting things here too about other uses of not just virtual reality, but augmented reality, which we haven't really touched on yet. But augmented reality does share a lot of the same features with virtual reality. It's more about marrying the virtual world with the physical world in a way where it's almost like an overlay

on top of the physical world. So one of the examples I saw was recently at Georgia Tech at their augmented reality lab, and they have um uh, they have an Auburn Avenue experience where the ideas you you put on or you hold up a display. You don't necessarily

have to wear one. It can be on a phone or a tablet or whatever, and you hold it up while you're on Auburn Avenue and it will show you historic views of what it looked like in decades past, which gives you this ability to explore a physical space in a new way. And that's something I've always been really excited about. But it is interesting also to see other elements using augmented reality. You have a whole sectional augmented reality therapy, which I wasn't even aware of. Yeah,

it's a surprisingly interesting field for medicinal uses. Uh, there's a phantom limb syndrome. If you guys are unfamiliar, it's basically the ideas a lot of amputees, the brain when you lose that limb rewires itself UM and they victim are victim but a UM, someone suffering from from being an amputee. They can have these phantom limbs, these phantom limbs sensations. They'll still feel the pain of these limbs that aren't there anymore, and there's no way to uh

to calm this pain or make it better. The limbs not there might might feel an itch on a limb that doesn't exists, so there's no way you can scratch that. You can't scratch it, you can't treat it. Um. It's it's been a thing that's been plaguing amputees since since it's been in the medical arena. And so this interesting thing. One of the ways that used to actually treat phantom limb was a mirror. They would take a mirror, and they would mirror it to the limb that you still have.

So let's say I don't have my left arm, but I have my right they would mirror it, so it looks like to your brain you have though it's in a mirror and it's very obvious. There's actually an episode of House Empty where they do this, UM and it's actually like pretty accurate. Um. But the idea is that you took your brain into scene into thinking that that limit is back, and then you can go about changing those feelings, uh, those those itches, those pains that you have.

And so there's this really really interesting um uh. There's an article in Frontiers and Neuroscience UM about using virtual reality, so you could you could be an emputee and in multiple ways multiple limbs missing, and they put on this virtual audi in the your brain being tricked into thinking that you now have these limbs back in this virtual world. Uh. It's it's been shown to be more effective than drug treatment.

Even there's they've cited a guy who he had been suffering from a phantom limp pain for forty eight years and this is the first time that it felt treated that he was having some relief, which is nuts is incredible to think that the immersion can be so convincing as to have that profound an effect on our psychology. As smart as we are, the brain is so easily tricked, yeh. I mean, we could talk about placebo effect in the whole little thing, or even stage magicians who very much

depend upon the things our brains. The shortcuts that we take where we make assumptions that we have perceived something even if we haven't actually perceived it. I mean, there are whole businesses, both legitimate and otherwise, that are based on that premise. Yeah, exactly. There's another one. There's a burned patient treatment that Loyal University Hospital has been using. UM. And this one's interesting because they basically, uh, it's a VR game, but it's like a VR FPS, but in

this case, it's like a first person snowballer. I guess like the ideas it's a it's a it's a game where you throw snowballs and you're in this arctic world. And it's not like the graphics. I was watching the video of it. It looks like PS one maybe early PS two er graphics. UM, But the sensation of being in this world, that's all ice and all snow and you're throwing snowballs at penguins and it's happy and you're having a good time. They'll put this on patients while

they're receiving treatment that's usually intensely painful. Um. Some of the treatments for for burn patient victims is like skin stretching is one thing which is painful without you know, having burn woods. Um. And consistently patients site uh, not feeling the pain nearly as much. Some of them even have a little bit of fun during it because they're

playing this game. UM. And it's again amazing. It's that this one little thing, even with the graphics that aren't that realistic, you have the sense of a cooling sensation of feeling that burning sensation for a moment. Yeah. I've seen that sort of response from other VR like researchers. I've talked to them and they say, yeah, there are times where you can have graphics that if you were to look at this on a screen, you'd say, well, that's not convincing at all. I mean it looks like

big polygone blocky graphics. It's not that far off from the pterodactyl experience I was talking about. Maybe you know better than that, but not so much that you'd say it was a you know, a quantum leap forward. But they'd say this is this doesn't matter. If you spend enough time to acclimate yourself to the virtual environment, your brain just starts to make those assumptions and starts to

actually produce those kind of reactions. And typically speaking, if the latency is low, because that's one of the big issues. The latency is low so that you don't have this disorienting feeling as you move around, takes about thirty seconds to a minute to really start to get acclimated. Yeah, it's amazing. It also shows how plastic our brains are that we're able to add that quickly. Yeah, there's a

there's another um Uh. Surgeons are using VR simulators even so back in the day, you'd have to test cadavers, which you're only gonna have so many of um that can be tested on so many ways before we have to move on to a new one. Ideally hopefully you're not testing on patients live, which you know was happening way way back. Why a lot of scientific progress made in that unethical science. Um. But so now we have VR systems that are super realistic um that surgeons can

use can use to train. And it's an interesting This is sort of related to that there are surgeons in Florida that they would find before surgery, they would have the surgeons play super monkey Ball, which is, if you haven't played it, super monkey Ball is a highly frustrating game where they the old wood game Labyrinth where you put the marble in the maze. There's holes all over to move the marble. Yeah, you have a couple of axes of tilting that you can do. Um, it's very

similar to that, very very similar um. And they found that surgeons who played just twenty minutes of this before surgery had was steady your hands much much better spatial reasoning and abilities. Interesting uh and again like very very little of practicing these reflexes in a virtual environment, giving them the ability to work much better while they're actually cutting you open and doing things inside you. Right. Yeah, So there are other ways of using VR in this

same sort of sense. In fact, one that I've seen that psychologists have used is immersion therapy for people who have post traumatic stress disorder or they have a phobia. They might not be able to cope in a real situation where something might trigger that that reaction, and so to get them more accustomed to it so they can build up to dealing with that in a real world situation.

Virtual environments have been used to acclimate people to that, so that because we have this feeling of immersion, if it's done well, just well enough. It doesn't even have to be fantastic, It just needs to be well enough to convince the brain that you're going through this the sequence of events that normally would trigger a negative response, that would be enough to get the physiological response going.

So if someone is afraid of heights and you put them in a virtual environment where it looks like they're on top of a building, their heart rates starts to go up, their blood pressure changes. I mean, they actually start to experience the physical reaction they would have if they were encountering this in real life. And yet you can reassure them that they are in a safe place. They are on the ground, they are nowhere near an edge of anything, and they can start to allow themselves

to acclimate and get used to this and explore. So the first time of patient might do this. They might just sit and look for a little bit and tell the psychologist. When they can't, they like, I can't do this anymore. Then they might get to a point where they're moving around, they might get to a point where they're able to walk to an edge. The goal of

this is never to cure someone. The goal is so that you can move up to a point where the patient feels comfortable enough to actually do some immersion in

in real life. So with the heights example, after you've done this for a while and done a few therapy sessions and UH and helped the patient be able to manage these responses, you might take a trip out to a building that has a few floors to it and go up to the top floor and allow the person to walk toward the window and and confront this in a real space, and because they've already done it in the virtual space, they're more capable of dealing with that.

I've also seen it for people who are afraid of airplanes, where they'll use it to have a virtual visit to an airport and they have the same experience of stress and anxiety that they would if they were to really go in a real place, And then the goal would be to ultimately take them to an actual airport so that they could confront this in in person once they feel comfortable in the virtual environment. Being a top seller.

It's funny because the the VR specialist I was talking to you said, yeah, changes in security have made it much hard order to deal with this because in the old days, you used to be able to go through security even without a ticket, Like you could just show up at the airport, go through security, and then walk to a gate because then you could you know Greek people when they got off the plane, right at the gate that they were at. I used to do it

all the time. You can't do that anymore. You have to have a ticket to go through security, so it makes it makes it much more expensive to treat these kind of things. You have to actually buy a ticket on a trip in order to be able to go through the security. So but that's, you know, a totally

different barrier than the VR stuff. The military also uses VR not just for training soldiers, but also as part of uh UH systems that allow people who are operating vehicles that normally wouldn't have a good line of sight to be able to see around them. In fact, some of the earliest headmale displays were made for pilots so that they could have a sixty degree view around the aircraft they were in, because there were parts of the aircraft that would block their visions and there'd be no

way of seeing through. That same thing with tanks. I've seen some tanks that have had um head mound displays and external mounted cameras with like a hundred eighty degree field of view so that you can get a real time They actually, uh, they stitched together all the videos so that you can turn around in place and see a three sixty degree view around the tank that you're in. Yeah, and that's you know, any any sort of thing that allows vision like physically is probably gonna be a weak

point on your craft. It actually reminds me of like Iron Man's heads up display. You know, that's his entire thing is being beamed into his suit, into his eyes. It's all video coming right back in. You know. If you had a big window face, he would be way more susceptible to all those. Yeah, and if any external cameras went dead, then he'd be blind. Yeah, exactly. Yeah, it's a it's it's certainly an interesting approach of using

that same technology for something different. As for the challenges of the R Well, the earliest one was that perception problem I talked about where we had Hollywood telling us that VR was going to be the future of interacting with machines, that you know, you weren't going to use a keyboard and a display anymore, you were going to be in the computer program. My favorite is any depiction of hacking where you would actually walk through amaze. Yeah, he turn turn, take a right, and then you see

like a skull and crossbones. You're like, oh, firewall and you turn around. What the other way? Do you think? This is not the way this works. I got to punch this firewall out of the way. Excuse me when I punched the firewall in the face. And you also have to wear a really off like you have to wear like a hot topic costume. Yeah, you have to do that. Also, just just in case you ever do enter an antiquated system, just know that no matter what

the password is, it doesn't matter. The first two guesses are going to be wrong and the third one is gonna be right. Yeah, third one. Ah, that's what it was his birthday and his uh dog's name. Yeah yeah, So use a strong password generator. Is the moral of that story. Not your birthday and your dog. No, don't not not my birthday and my dog either, because it is my birthday and I know the trick. Um late

and see again. Another big issue that is the thing that can make you feel pretty nauseated pretty quickly if it's not solved. And then that interface like we're talking about before, if you don't have that fade into the background, it can really become a barrier to immersion and interactivity. So here comes the last little bit of this episode. It's been an epic episode, Ruben. What do you think you think is the consumer market ready for virtual reality now?

I mean I remember in the nineties when it was first coming out, that the whole virtual boy era. I remember all of that really well, and it definitely wasn't ready yet. Do you think it's ready now? Um? You know, it's funny. I think we're I think we're in the early days of it becoming a real thing. Um. I think Iculus riffed the Kickstarters apport behind that. I think it's proof enough that there's a market for it, that people want it. I mean they made multiple times more

than what they were asked. Yeah, you know, it was it was one. It was for a while, it was the most successful kickstarter. Uh. If you compare what they were asking for versus is what they got, it was ten times the amount. And I think it's a great it's a great idea, especially the technology is getting there. As far as the visual thing, you can actually they make little headpieces. Google has a cardboard box. You can put your smartphone in it and get steroscopic vision to

your cell phone. Like we have tiny tiny five inch slabs of l C deer, amile D whatever it is UM that are strong enough and fast enough to uh to produce these virtual images. So I think a lot of our technology for the visions there. The biggest hindrance I think is going to be cost. Like the Oculus rift is actually a fairly reasonable price wise, but then

you get omnidirectional treadmill thing. Even if it's just that that teflon one, no moving parts really, UM, that's still fairly expensive and it requires space, which means that's another expense because you know, we don't think of it as space equals money, but not everyone has a place big enough to have a standing, you know, pet stole that

is just there as an interface for computer game. The first connect for the Xbox of the new connect connected to I guess has a much wider field division from up close, but the first one didn't, and a lot of game reviewers ran into problems in their tiny New York apartments of like, I can't I can't play this game apartment. It cuts off all of my limbs because

I'm too close to the camera. But if we even just think about, like so Sony's Project Morpheus, which they have a lot of the ecosystem already there, the PlayStation for a camera. So not only you need the headset the Morpheus itself whenever it becomes commercially available, if it does, you need the camera, You need the headset, You need at least one probably of the two two of the

move controllers who can map your hands. We're not even going into uh, your lower body, and that's that's a lot of and the PS four so we're talking about, you know about a grand buy in just for the very very basics yet yeah, and not even a game for it. Um So, I definitely think that we're sort of on that us of doing things that are reminiscent of the crazy stuff that we saw in the nineties and will be exactly that. But we're getting closer. But as far as like mass adoption, I feel like i'd

give us five to ten years. Yeah, I I agree with that. I think I think on top of that, on top of all the challenges you mentioned, there are other ones that we don't think about as consumers that are on the back end, like developing for this system, because developing a game that actually works in this environment and is fun and functional, that's a that's a tall order.

I mean, you're really talking about the difference between say, shooting a movie as a regular cinematic experience and shooting an actual three D movie that was intended to be three D the entire time. And we talk about ten a DP and everything. When you're do in virtual reality, you're splitting, you have to split your resolution into so that's a crazy horsepower to can maintain the same fidelity

we have in a single vision. Sure. Yeah. Yeah, So there there's some big challenges that we're going to be facing technologically as well as just economically. You know the idea of doing the system where it's affordable, do we make it a loss leader where we're going to try and make it up in the content we create. If we don't sell enough of them, are we going to be able to create the content that will let us

recapture the costs? I mean, there are a lot of questions to answer, and I don't think we're quite there yet. I think we are on that that edge where excuse me again, I don't think we're right we're there just yet.

I think we're around the edge where it could go either way, where we could either see some really promising development and thus that pushes consumers to really get excited by it, or we could have these challenges we've talked about show that we're not quite ready, that there's still some pieces missing that we need to solve first, and then maybe in five to ten years we'll have the pieces there where this will become at least a form of consuming entertainment. I don't. I also don't think it's

ever going to become the predominant one. I think it will always be a section of the gaming community. I don't think it's going to be like I can't imagine that the console, the next console in ten years, with the PS four and Xbox One, I've completely gone through their life cycle, is going to be a head mounted system. Yeah, I think we'll see it asn't been supported maybe in the next generation, as sort of like it's going to

support this automatically like three D was supported stuff last gen. Um. I do think gaming is probably gonna be the herald of this by all means. But we see like Google Glass with the augmented reality that is not necessarily for gamers at all. There's some there's some filmmakers that are exploring the ability of of making stuff specifically for people to watch while wearing the oculus where you you can

look around an entire scene and see everything that's going on. Now, that of course, presents its own series of incredible challenges to the filmmaking industry. Film the three sixty degrees every

time everywhere? Have you not see the set somehow not see not see like like the the teamsters lounging in the background smoking cigarettes and drinking coffee, or or in the case of h you know, um uh other forms of media just do you how do you make everything interesting enough to look at, knowing that someone is always going to be looking at that point at any given moment. Not everyone will, but you know, it's just like theater

in the round. If you if you set up a theater in the round, you have to assume that someone is looking at every single part of that scene at any given time. You know, even if that's not supposed to be the focus of the scene, someone's looking there. So you've got to be present, and you've gotta be ready to go, and you've gotta make sure it looks good.

And UH has its own huge sets of challenges. I think when the most terrifying indicators that it is going to happen is Facebook taking them Facebook buying oculus, I'm taking too massive steak and something that used to you go, oh, why would why would Facebook want this? They firmly believe in the integration that it has and you know, they have a alien users. There's only you know, after a while, like growth plateaus, and then you think, all right, well

we have to diversify. If we only concentrate on this one thing, we can't grow year over year. And if we can't grow, investors aren't happy, and the investors aren't happy, then we lose money and then that's a downward spiral. So this is a you know, this is a big bet for Facebook and UH personally, I hope it pays off, because I think it the future could be really incredible with us uh and uh, I certainly, I'm I'm enthusiastic about it, even though my own personal experience was a

little uh, you know, stomach turning. But again, I didn't have enough time to really um acclimate myself, so I think after a little while I probably would have gotten used to it. If I can be a virtual sandwich artist and I go home, I'll be so happy. I was going to go right back to papers. Please, Ruben, thank you so much for joining me today. This is fantastic conversation, pretty long episode. I hope you guys enjoyed it. Ruben. Where can people find you? Oh boy? Um, so I

guess uh. I'm out on Twitter, um at robots r O O b O T s um heads up, very little things. A very few of the things I'll mention here are safer work, including my Twitter. Um. But also you can find me at Dad's garage. That's where do most of my theater stuff here in town in Atlanta. So if you're in town by all means comes wing by check out a show. Definitely do that. Dad's garage is one of my favorite places in the city. We're

doing a murder mystery tonight that's Star Wars themed. Just in case you're you're concerned about our nerd credentials, Trust me, the people at Dad's garage are probably some of the nerdiest prints I'd have was. I was painting Karelian blood stripes on pants last night for Han Solo wonderful so um yeah. Um also ed Tom and Reuben dot com. It's a little dicky say, I threw up for me and two guys from Dad's. We are doing sketches and podcasts. Um.

We have a podcast called Welcome to super Basement. Um. Trying to think of how to truncate the description. How would you describe it? John? Welcome to Super Basement is the story of three friends who inhabit a basement that is so huge that it incorporates everything in reality and fantasy and science fiction all in one. It is completely improvised.

It is approximately a half hour per episode where the three friends discover some form of adventure and then go on it and maybe there's a conclusion and inappropriate jam. Oh yeah, No. It's very much not safe for work, but it's incredibly hilarious. I absolutely adore the show. I highly recommend checking out Welcome to super Basement. If you are of an age to appreciate not safe for work humor, you can subscribe on there. Yeah, that's fantastic, Reuben, Thank

you again. Thank you guys. I hope you enjoyed this episode. If you have suggestions for future episodes, or maybe a request for a special guest host or interview, let me know. Send me a message my email addresses, text stuff at how stuff works dot com, or drop me a line on Facebook, Twitter or Tumbler. The handle it all three is tech Stuff hs W and will talk to you again really soon for more on this and thousands of other topics. Because it has to have work dot com