TechStuff Classic TechStuff Hacks the Kinect

the peripheral for the Xbox platform. And uh, back in two thousand twelve, that was still a thing. It's largely not a thing. I'll talk more about that at the end of this episode, but in the meantime, let's rejoin young Jonathan Strickland and his plucky co host Chris Palette as they talk about the Connect. Jonathan and I had been talking about things that you can do with other things. Yeah, okay, that was a little oversimplified. Basically existing technologies that could

be repurposed for other means. We thought it would be fun to talk about a series of different kinds of technologies to not just uh not just say game controllers like the connect. Um So you know, this is one of those that that we're doing, and we thought this one in particular made an interesting topic because so many people have repurposed Microsoft's UM Motion sensitive controller UM and it's uh, actually for very simple reasons. It's really inexpensive

compared to other ways to do this. In fact, this is very readily available. This is something that we've talked about this in a previous episode really because back when we were talked about virtual reality, Yeah, virtual reality, Uh, you know, for a while that was like the Darling phrase and technology from the late nineties. Do you remember that era where you could shoot the pterodactyl and get

incredibly dizzy through the latency is choose? And the funny thing about virtual reality is it still exists, but the phrase has become so dated and associated with that big headgear and the giant gloves that you had to be wired into or the holiday. Yeah, there's a stigma there, and and the ideas have become much more refined, but they're sort of augmented reality or just virtuality, virtual environment

sometimes of environments. But yeah, the connect is it is one of those things that sort of bridge the gap between the real world and what you see on the

computer or on the TV. Right and like you said, it's it's readily available, and it's an expensive That was the big problem that virtual environment scientists and and and pioneers were running into is the fact that they could build the software, but designing the hardware was a lot more difficult, because you're talking about an incredibly expensive process to prototype something and then build out a working version

of it and then go into mass production. If you ever want to go beyond just a laboratory experiment, that's that's hard to do. And so virtual environment folks often end up looking to the world of video games to repurpose the equipment that is made as just a form of entertainment so that they can use it in their own projects. So we've seen this with things like the we controller, uh, the Sony move controller, as well as the connect. It's one of those very easy to get

your hands on pieces of technology. Yep, yep, um. Yeah. Those early uh, those early devices required you to uh put on glasses or wear a helmet that had a display that you could see so that you knew what you were doing in this virtual world. You had to have some kind of something on your body, uh, often gloves or some kind of sensors. That would relay information to the computer generating this virtual world. Usually it would be a combination of something you were holding or wearing.

Plus the headmild display would have some tracking in it as well for head tracking. And in some cases you'd have to be on some kind of virtual environment too. You'd have to be on a treadmill or something that would track your movements or would make it seem like you were walking in this virtual environment. And as you can imagine, this is all really expensive. Um. The newer systems that that we use now are used infrared or

or cameras regular cameras to capture where you are. They map out the room, they know and they know what's going on in your actual physical environment where you are. And um, you don't have to necessarily be wired in like you used to be a lot of it's also using Bluetooth or or WiFi to communicate, Uh, makes life so much more easy. And plus we're talking about consoles

that are much more powerful. Though then the computers were back in the old you know, even in the days when people were building virtual environments and they're building those first applications. The consoles that we have today are in many ways comparable or sometimes superior to the equipment that

they were using back then. So let's talk a little bit about the connect, what makes it work and how it is able to detect depth, which is I mean, that's the that's the key to many of the connect hacks is the fact that the connect is able to map out a three D virtual environment based upon a physical space. Thankfully, I found a really cool diagram in an article by Jason Tan's called connect Hackers are Changing the future of robotics. Um, the connect is a wired device. Um,

But attached to that wire are some actually pretty simple electronics. Um. It's got an array of microphones, right, video camera, video camera, color camera. As a matter of fact, Um, it's got an infrared emitter, yes, which I will explain what that does in a minute. Um, and a sensor as well to pick up what that emitter is, uh, the reflections of that of the stuff that the emitter is spreading out. Yea. Yeah. It's also got a depth camera, which essentially creates a

map of the room. And it's got a series of well, it's got a tilt motor in it yea, to allow the device itself to change its own uh. Perspective really, which is very useful and lots of different applications also, so you can get go connecting and guess, okay, no, it doesn't. How can the connect detect huh? A three

dimensional environment? So you know, with a three D camera, the way we would normally make a three D movie is that you would have a camera that would have two sets of lenses set apart from one another in such a way so that two different sets of m just could be combined. Uh. And so when we watch it, our brains do the work and put it all together,

and it makes the illusion of three D. Right. The two cameras are spaced about the average with of human eye, so that that it creates that illusion of depth because it it takes two images um set apart and about that with and it makes us believe. It makes our brains think that it's in three D right, right, And each each set of images goes to one of our eyes. Right.

So that's why people who have vision problems with one eye or what often have difficulty perceiving a three D film because for it to really work, both eyes have to receive their individual sets of information. That's why three D glasses are the way they are, it's to filter out the other set of images so that only one set goes to each eye. Well, that's not how the connect does this. The connect has that emitter and the

sensor or the depth camera to do this. And what's going on is that Imagine that you've got your your living room set up, okay, and you've got your connect connected to your Xbox three six so imagined. Alright, so you're standing in front of the connect. How does the connect tell that you are a three dimensional person? Because remember when a regular video camera, you're getting a flattened image. That's that's the real problem with with a basic cameras

that it flattens everything out. It does not detect depth. The only que we have visually is essentially the uh perspective, which is why and the Lord of the Rings movies, when you had characters standing really far away from characters who are really close to the camera. That created the forced perspective illusion that one character was larger significantly than

another character. Uh happens all the time in films. Well, the way the Connect gets around this is that emitter that I infrared emitter says out essentially a grid that is made up of infrared light. Now we cannot see infrared light, so we do not perceive this, right, It's outside the visible spectrum for human beings unless you're an alpha a dwarf from advanced stages in the Dragons, in which case you might be able to see this, but you're probably not playing Xbox if you're one of those,

unless it's skyrin anyway. So the this grid gets projected over the physical environment. Well, when the grid encounters a physical object, it's deformed. The grid itself is deformed, not the object. The object stays the same, otherwise we would have a lot more to talk about with the connect. So the grid deforms when it contacts comes into contact with a physical object. The depth camera measures the deformities that are within this grid and can interpret that as

objects that are a certain distance from the connect. So as it tracks the changes in these deformities, it text player movement. So when you move your arm, that grid is going to change, and the depth sensor is going to to capture those changes, send that information to a processor which then interprets those changes as various commands. So, for example, if it's a game where you shove your hand out in order to make a character shoot a gun.

Then what happens is the uh, the grid will be deformed as your hand goes closer to the camera on the or the to the infrared emitter. Really from the connect, that grid deforms in a certain way. The depth sensor camera detects it and then sends the appropriate command or the or the data to the processor which process of the appropriate command for the game. So that's the basics behind the depth sensing technology of the connect. There are

a lot of other sensors out there. They use similar technology, not all of them are using that grid approach. Because one of the um downsides to this is that the closer you get to the camera, the less accurate this depth sensing technology is because the grid is of the squares within that grid are a certain size. So as you get closer and closer, you are deforming less of the grid with your various movements because your your movements are taking place more within the empty space of the

box as opposed to the various intersecting lines. If that makes any sense, No, it does, Um and uh it's funny because this this technology is we think about it. This this this technology is designed to be used in the living room as part of a game system. Um. It's it's designed to be inexpensive so people can afford it, but still to be sophisticated. Um. And that's why. Um and it was designed by Microsoft, which is why people

want to hack it. Um. Actually, I think that's probably a lesser reason, but yeah, I'm a motivating factor for some people, I think. I think the motivating factor is mainly that it's a cool technology that can do a lot of stuff. But the fact that Microsoft is the one behind it may give some people a little extra motivation that that. Yeah, that's that that slightly turned up smile and right. Um, no, it's it's available in many

many places. Um and uh you know that's that's sort of what provided the impetus for sort of a competition to see who could hack into it first and make sense of the the communications. Right. So the Connect launched officially on November four, and then immediately as when we're recording it, right, and immediately there was this this challenge issued. It was like a bounty essentially for the first people to to really hack the Connect Microsoft. Uh. The initial

reaction from the company was negative. The come be the officials originally when news was breaking about people trying to hack the connects, said that they were, uh, they were not in favor of people modifying their products, and in fact, they would work with law enforcement officials to try and keep people from tampering with the connect Yes, but at that point, if you think about it, the the Xbox three sixty had been out before, um, before Nintendo's Wii system,

and a lot of people have dismissed the Wii as being a lightweight, didn't have high deaf uh you know, not not full ten eight high deaf, and it was seen as a game system for casual gamers, which among amongst a certain subset of hardcore gamers, that has a stigma right for a hardcore gamer for some, not all because I'm a hardcore gamer in many ways, but I don't share this particular opinion, but for some hardcore gamers, they view it as it's not a quote unquote real

gaming system because it doesn't cater to their particular gaming preferences, right, right, And you know, we've talked about this on on multiple podcast so I don't want to get into it in in great depth, um, but I think that uh, the success of the weeze control system using the motion Wireless

Motion Control system, UH, surprised both Sony and Microsoft. UH. And then Microsoft was the first to come out with its own upgrade to the three sixty, which you know wasn't actually the three sixty was actually the Connect itself. UM and we we heard about this um as project at all back in the in the early days of

the system. But then they released the Connect and it was immensely successful from the get go as far as the the Xbox three sixty because it gave you all kinds of new ways to interact with the system and the games. It's sold like hotcakes. It became the fastest, uh fastest ever selling device to get broke all the records at the time of its release. Now I should add that before we get a ton of listener mail

and responses to this. There have been some criticisms about the Microsoft support of the Connect, saying that perhaps there aren't quite as many games to support the device as people would prefer, but it's still been an incredible resource for hackers. Yes so so, maybe not as much for

hardcore video gamers. Although there are some great Connect games out there, it's just that some connect some people say like, okay, well, the added benefit of the Connect hasn't really been fully realized in the game space, but it is it's an

incredible tool for hackers. Yeah, Microsoft, Microsoft came out with a really nifty device in the Connect, And the only reason I mentioned that bit of background was um I think it may have played into their Microsoft's response to the hacker community doing this because, as Jonathan mentioned, a moment ago, they said, no, we this is a proprietary device. We do not want anybody mucking around with it and

hacking it. It is designed for the Xbox three sixty And then a very short time later, like basically they had the weekend to think about it. It was so so it launched on November four. They shortly after launched discussed the idea of we do not want people hacking with this. But on November nineteen, there was period of time right there was a there was a developer for the Connect who specifically said that they left the Connect open by design, it was meant to be a hackable device,

and Microsoft got behind that at that point. I think I think in part they saw what the public opinion was, and you know, they probably also learned from other companies, maybe not mistakes, but their approaches to this same sort of thing, like a famous one Sony's PS three. Yes, the original run of the PS three allowed you to uh to load up a Linux based operating system to the PS three, turning it into a very powerful computer.

And if you've got a whole bunch of them together, you could make essentially what was equivalent to a supercomputer running on this alternate operating system. But then in later iterations of the p S three, later models, Sony removed that they first they began to send out patches to the firmware so that it would prevent you from loading in that operating system on the older models, and then the newer models just had no support for it at all.

And Sony caught a lot of flak from a lot of different hackers and computer scientists saying, you've just taken away a tool that was incredibly powerful and affordable that would have done a lot of good, and now we can't do that anymore. And clearly the military, the military was using PS three's as well in this case because they were they were inexpensive compared to the other kinds

of supercomputer quote unquote supercomputer material or hardware. They they were looking into um and it was easy to do so. And that's why the hacker community with Microsoft now blessing, they've said, you know, okay, let's do this. Yeah. I think I think I think that I think Microsoft realized I said, you know, this is First of all, it's a losing battle because hackers are gonna hack. Second of all, it's better, it's a better pr move if we support

it then if we don't. And third they may have just again, they may have just honestly rethought it and came to the conclusion of this makes way more sense. First of all, we're gonna sell way more connects if we actually support this community, and you never know what can come out of it. And some phenomenal stuff has

come out of the hacking community. And uh, and some of it came out almost immediately, like like within within a couple of weeks of the connect becoming a product that you could actually buy off the store shelf and bring it home, you started seeing some really innovative hacks using it. Yeah. Now, also just as a mitigating factor three a or three and a half on your scale. Uh. Microsoft also realized that the proprietary information inside the connect

is remaining proprietary. The people the things that the hackers are are doing basically just use the signals from the connect. We found ways to use to write software for and use the signals from the connect, and they're not actually getting into the stuff that's truly proprietary about the connect. And so Microsoft went, We're really there's nothing to be afraid of, um, which is very cool. It's not like it's not like hackers are trying to create a cheaper

version of the connect and sell that or reverse engineer it. Yeah, they're not, they're not doing that. But let's talk about the stuff that they are doing. There's so many amazing things, all kinds of things. So they fall into two general broad categories. Usually there are quite a few that fit into very broad categorizations, like user interfaces would be one being using the connect in various ways as a user interface, which is not a big surprise. I mean, that's the

original intended use of the connect. It's a user interface for the Xbox three sixty. But there are people who have designed it to be a user interface for lots of other stuff, everything from a computer where you're just using gesture controls and voice controls to interact with the

computer two robotics. I showed a video to Chris and our producer Tyler just before we went live on this podcast, talking about a project that created a robotic trash can and it was a very clever approach and and it wasn't It was using to connect in a way I just did not. It didn't occur to me it Uh. Actually it mirrors what happened with the Microsoft connect Um in that, uh, people saw a challenge and they rose to meet the challenge. It was a based off a

commercial for what gum mints? Mints? Okay, it was commercial that the idea of being that it shows a guy at a typewriter and a typewriter kids, ask your parents because it's got a typewriter and pulling out a sheet of paper from the typewriter and crumpling it up and throwing it behind his back and does this does this a couple of times working on a project, and he's just coming up with drafts that don't match to his expectations, so each one he pulls out and throws away and

then he then he takes He's very inaccurate, right right, So yeah, it cuts behind him, and there's just all these little crumpled pieces of paper on the floor. Then he eats a couple of mints, crumples up the next piece of paper, throws it behind him, and then the trash can moves to intercept where the paper is coming from and catches the paper. Right. Yeah, it says that it will make you The mints will make you sharper. Actually seems like they should. He's apparently been feeding the

mints trash can. But yeah, so so commercial. Some some some robotics and computer scientists types decided to try and recreate that moving trash can. So they built a robotic base that had three wheels on it that would allow the the base to move in any given direction at a pretty good clip, as as the videos shows. Then they built a essentially a trash can chassis that fits over that, so so it disguises the fact that there's

this base with wheels on it. Just from a casual glance, it looks like a trash can that goes all the way to the floor. There's actually just a little bit of clearance so that the wheels can move around. Yes, yes, they obviously were very careful in their measurements because it fits directly over the robot base and you cannot tell just from a casual glance that there's a robot inside the trash can. And I'm assuming they probably made a

false bottom too to cover the robot. They didn't show that part, but they they did show that they were actually machining these parts like they were using us. Yeah. So, uh, the the remote control aspect, they hooked up to a computer and they wrote some software and they used a connect sensor. They mounted it to the wall. So what the connect censor does is it essentially makes a three D map of the room, right, so it's able to

detect depth and and direction and movement. And what they did was they created a program so that when you tossed any small object through this space, the connect would detect the arc of that particular object and plot out where it was going to land, send a command to this robotic trash can, which would intercept that path and catch the object. So this has to happen really fast.

It has to detect the arc, plotted out its course, send the information to the robot, and the robot has to have time to move to the right location to catch it. And we were watching the video is probably on average, the trash can is probably moving three or four ft, right, maybe three ft let's let's say three Yeah, So it's um but it's still pretty impressive to to watch them toss various objects into the air and the trash can moves to intercept and catch it, and you think, well,

that's kind of a ridiculous use of the Connect. I mean, but it shows that the three D mapping technology uh within this device can be used for some really really creative um uh projects. And there I've already shared the video of this trash can on our Facebook page, but I'll be sure to tweet it out to uh so you guys can You may have already seen it, but I just was so amused at that particular use of

the Connect. We have a bit more to say about the Microsoft Connect and how it works, but first let's go and take a quick break to thank our sponsor. VS Robotics engineers have built this robotics trash can robotic trash can for fun to meet the challenge that they perceived. It wasn't even a real challenge. With perceived challenge of doing this because they saw it on the commercial I thought it was cool at least that that's what we

can guess. We don't read the it's it's produced out of Asia, and neither of us read the language that they wrote everything down in. But that's what that's our interpretation. Yeah, so and okay, so they've they've it's proof of concept. If you will, it can be done. But what if you had a a real challenge. You've had somebody who uh needed help. For example, um, they were suffering from dementia or some kind of physical impairment that kept them from interacting with their world in a way that that

you know, the typical person would. So let's say, uh, they have this h a chair based on this technology, and somebody who might have trouble sitting down and the chair moved slightly to the left to make sure that they're going to sit squarely in the chair instead of missing it and hitting the floor and needing assistance getting up.

Sure you could, and you could use the technology developed by Microsoft to connect to and this this software and the technology developed by these guys apparently just playing around and use it for something practical in the real world. So honestly, I just messing around on the top of my head as you were describing that, I was thinking, Yeah, it could could be silly, but it could be used

for something really useful. Yeah, and that's just something I you know, well, and there's there are plenty of other examples of of this being a user interface, for example, like the one that I talked about earlier, where you're navigating a computer system using UH gestures, kind of like the old Minority Report approach, except instead of being projected in front of you, it's you know, it's on a screen, but you're you're moving your hands to just your control

whatever it is you're looking at. UM. That's AH. And and of course that's something that Microsoft itself has taught about the possibility UH that you know, we would see a connect like sensor being built into computers in the future, and their versions of Windows would support that. And if you look at something like Windows eight, it's not hard to imagine that just based on the metro layout that

they have, and it does already work on computers. You can use a connect for computer gaming as well as for the Xbox three sixty. So it's you know, hey, we're getting close to that word that you haven't used in some time. Yeah, there's another cool use of connect I saw was the gestural spatial mixer. Did you see this the gestural spatial mixer mixer? Yeah, it's a student. Unfortunately I wasn't able to get the information about his name because the blog I was looking at didn't didn't

list it. But it was this graduate student who had designed a program where you could play multiple tracks for uh. Let's say. Let's say let's say you've created at a music track and it's got various channels in it, okay, and and you've got a multi speaker set up. So he was using I think an array of speakers where there's something like six of them. He had a connect hack where he had designed the software and he was using the connect two manipulate the music so that you

could isolate channels. So if you just want to listen to one channel within that entire music track, you could you could even move that channel from one speaker to another. So let's say that you know, I just want this to come from the left side, I'm going to move it to the two speakers furthest to the left, and it's going to not even beyond the four speakers on

the right. Um, you could do that and you could change the volume, and you could make it sound like it was the music was coming from nearby or far away. It would add an extra effects for that kind of stuff. So he was here far so he was doing all of this in real time, just manipulating the music physically.

So it's almost like being the conductor of an orchestra, except of course, you know that certain things would always play out the same way, whether he wasn't affecting the the speed of the playback or anything like that, but volume and some other effects as well as where the music appeared. He had that all in his demo. Also, the flutes are more likely to be in tune. Uh. Sorry, that wasn't very nice. Whatever you're you're a drummer. You cannot make jokes about other musicians. I'm sorry what I

was drooling on something? Anyway, Yes, I can we all make jokes about one another anyhow. Um yeah, I mean that's that's that's the idea of being able to do that is is fascinating. I've seen, uh, quite a lot of of those kinds of things, but they're typically done with infrared light or you break a light beam to make that happen, and the idea of using the connect for that purpose. Yeah, and the and the light beam stuff tends to be a lot more limited. You know,

you don't have the versatility that you would with a connect. Uh, there was. We actually have an article about some connect hacks as well as an article about how the connect itself works. So you should go to how stuff works dot com if you want more information. The article on how the connect works is helpful, and the hacks are kind of fun too. One of the ones mentioned talks about, um, well this this isn't actually in our article. This is one of the other ones I discovered was a connect

hack that was hooked up to Tesla coils. Actually, I think this one is in our article. Um and the Tesla coils. If you don't know, these are things that create high voltage electrical currents. You could change the frequency of those currents. And it's this sparking thing that you often see in old science fiction movies that has a standard part of a mad scientist layer. It's not the Jacob's letterer, but was gonna says, that's more of a

jacket's letters. Um. But now the Tesla coils, it's this, uh, you know, this massive thing it can be a massive thing that shoots out deadly sparks. Um you don't want to get hit by them. We've we've seen groups like

Architect that used Tesla coils to create music. Well. At maker Fair the UK, a guy named Tom Wyatt headed up a team where they showed off a connect that they hooked up to Tesla coils and they used the motions that they would The connect would pick up motions that Wyatt and his team would make and translate that into different voltage frequencies for the Tesla coil, so you can make the Tesla coil emit electricity at different frequencies, which actually has a different sound to it. That's how

Architect makes music with these things. They program in specific frequencies so that they can create tones. So one of my favorites is they use it to create a cover of the Doctor Who theme, which is pretty ofesome. I've also seen a really good one where they used a special suit they could wear which allows the person to actually be hit by the uh the electrical currents without frying uh. And they did the Imperial March from Star Wars, so it looks like he's shooting electricity from his hands.

As the theme is playing really subtle anyway, Um, that was a really cool one. Uh. They the article also talked about one from the University of Washington. They used the connect when in the use of robotic surgeries. Now these are surgeries that are not done on robots but by robots, so so sometimes that's far more common than it used to be. Yeah, this is when you're do is tell usurgery right where the the actual surgeon is in one location and is controlling robotic implements to perform

a physical surgery that could be half a world away. Well, the use of the connect is meant to do things like create a virtual map of where a person's organs are. The idea being that when you're using a robotic tool to cut a person or whatever, whatever the surgical procedure is, make an incision, should you well that would be one

of them. But should you reach, should you start to approach the area of an organ, it would send the signal because the connectors mapped this out in the text, that you're getting close to one of those areas, send a signal that would give you some sort of haptic feedback, so you would actually feel a warning saying hey, you

don't want to. You know, you gotta be careful because now you're approaching a vital area because again, when you don't have when you're when you're distanced like that, it's a lot harder to tell, you know, how close you are to something like that. So that's a pretty interesting use of connecting or something that was used for video games could potentially be used to save lives. Yes, yes, definitely. So um you realize that this is uh actually pretty easy to do when you get right down to it.

I was looking at um uh articles about how to hack with the connect and basically all it requires is a computer and a connect. Um you might need a power supply for the connect if you're hooking it up to a computer if you don't have one, because they don't necessarily come with them if you're buying the one that comes with the the Xbox three sixty. Um, but

you know they're there. There's a wealth of information online and you can look it up in a search engine for connect tax and you can find the software that you need, and it depends on what you're you're doing with it. But there's already uh some pre written software available out there by other hackers who have done maybe something very close to what you're planning, UM, that you might be able to to get so it it Uh you know, there's really not a great big boundary to

getting involved with it. Um. There there are plenty of things like robotics and things like this. But one of the really cool uh applications of hacking connect technology I've seen is uh someone who's using uh uh you know, they're scientists using the connect to do geographical measurements on UM on bodies and space and on on glaciers. UM found an article by Adam man Um about Ken Mankoff,

who basically was doing scans of of caves using the connect. UM. Now he this is this is one of those times when the connects limitations come into play because, Uh, if if the sensor were or sensors in the connect were more or were designed to scan larger bodies instead of your living room or larger environments, let's say that um than he wouldn't have to do quite so many measurements with the connect. But um he has found that it is far cheaper to buy a connect for the computer

than it would be to rent the sophisticated equipment. A lot of times they use a light detection and ranging which is light ar UM and they can use uh, you know, they can use these over you know, much longer distances, including miles, you know. But the problem is, um, you know, it costs ten to two hundred thousand dollars

for light ar as opposed to bucks for a connect. Now, if you guys have seen the documentary Prometheus, you know about the use of of light ar to map out things like underground caverns where they had these probes go in and use light ar to create a full virtual map of that environment. Uh. That of course is a movie, not a documentary. I make that joke all the time, but just in case you weren't familiar. But but yeah, I've seen other uses of this tech for the same

of the purpose, using them for rescue robots. Essentially, you would send let's say that you have an environment that is dangerous and there may or may not be people in the environment, or perhaps some other thing that's in that environment that we should really try and get close to,

but we don't know the nature of that environment. How dangerous would it be, uh, sending in a robot that's got to connect that can do this three D mapping could be very useful because it gives you an idea of what the conditions are within that environment and whether or not it is wise to send in a human to try and navigate it. Um. So yeah, I mean that that particular uses everything from geology to rescue operations.

Younger Jonathan has apparently made the gesture control to put the episode on pause so that we can take a quick break to thank our sponsor. Will be right back the article also that I read also um uh talked to uh a scientist named they are mob Chovitz who wants to put uh, you know, devices maybe not necessarily the Connect, but devices like the Connect on on space exploration robots and have them fly in close to asteroids

to do a map of the surface. Um. The applications for this device have really spawned, uh, quite a lot of speculation and interest among the hacker community, among and amongst serious thinkers. Then another, the two worlds don't converge. Well, no, they not necessarily, but I mean people uh yeah, I mean people who are doing this just for fun and people who are doing this for very serious reasons or

or both. Um, and I'm sure they inspire one another and I'm sure that when I mean based on on what the limitations of the Connect and what it can't do, it wasn't intended to do these things. But wow, you know, he really really just didn't anticipate this coming, and it's great that it has. I have a couple of others I want to just really quickly mentioned. So one of them is a second Story Labs. They have started to use the Connect. You know, Connect has evolved as well,

like the firmware has evolved over time. Microsoft built in some facial tracking capabilities that were updated to the Connect. So the original Connect wasn't as good at it, but now it's gotten much better where it can track a person's face. In fact, that's caused some people a little bit of concern. But but Second Story Labs what they did was they took this capability and they used it for an interface where, uh, they can create the illusion

of a holographic display. Yeah. So you've got imagine that you've got a glass window in front of you, and to you, it looks like a three dimensional image is floating behind that glass, right, and as you move your head you can see the different parts of the of the image. Well, what's actually happening with Second Story Labs.

Is approach is they're using to connect to track your facial movements so that as you move, it can display a different Uh, it can it can alter the appearance of whatever the display is showing, so that it would show you the angle, like what would happen if you were actually looking at an object from that angle. So, in other words, it's not a true hologram. It's still it's still two dimensional image, but it's adjusting on the fly as you move around so that it looks like

a three dimensional image. So to your brain it might as well be three dimensional, which I think is a really creative way of getting around this whole glass is free three D thing. Yep. And it could also and I imagine it will. You know, we were talking about UM Minority Report earlier and in in the movie, Uh, you know they're walking around in interactive environments where people are talking specifically to you about things that they know

about you. Um. Maybe not necessarily that Again, you were talking about the facial recognition, they could do that, um, but yeah, you might start seeing um, shopping environments that are that start talking to you. Um, you know when they say here, well, you know, there are some displays that you walk by now in their motion sensors in

the display, but it's right there at the display. You can make an entire environment that knows where a person is and they say, oh, I see you're looking at the computers, Well we have these they've got these processors. Or you might have an automated salesperson talking to you about the benefits of one model over another because they know you're standing right in front of those, or you might just use them to help design better uh storefronts.

Like imagine imagine having a window on a on a street, you know, sidewalk, you're you're winding on the side while you're looking into a display window. You could just as easily use a connect if it had if it had Sophiska enough tracking capabilities where it can actually track where you are looking, and you could say, all right, well we tested out this one display using this particular layout, and this is where people were looking, and that's not why we wanted them to look at. We really wanted

to feature this other thing. So then we did the display a different way like this, and everyone was looking where we wanted them to. So that's the way we need to do it in order to promote whatever it is we're selling. That's just one simple way of that could be used. A connect could be used in the future. UM another cool hack that I thought was actually really really interesting and it has some really the the experiment

is interesting, the potential applications are pretty phenomenal. UH. Alex Brown and Brian Brown they worked together to create a virtual realm that would allow you to interact with virtual objects that could in turn control actual physical objects in your real environment around you. So what they did was

they used a programmable automation controller UH. And they had this programmable automation controller panel that had switches on it and when you turned on one switch and l e ED would come on, and when you when you hit the switch next to it, the LED would go off. So there was a physical switch that would turn that

LED on or off. Okay, So then they built a virtual representation of this console in a virtual environment, and by virtually touching the switch, it would cause the physical l e ED on the real console to come on, and then by touching the other switch, it would make the light go back off. So it's just like you're touching the physical controls but you're doing it in the virtual realm. Now, imagine, let's say that you've created a

entertainment complex. Okay, that is a virtual reality entertainment complex. So you, Chris, go into this virtual reality entertainment complex, and you are able to interact with this virtual environment, and as you interact in the virtual environment, physical things happen as a result of your virtual actions. So let's say it's a role playing game and you are playing in a really uh you're you're in the Arctic, and you're inside a room when you first start, but you

decided to go outside. When you go outside by opening the door, it activates a fan system that blows very cold air on you so that you get the sensation. So it creates that tactile, immersive environment. That's just one example of a potential application of this sort of technology. A lot of other ones would have nothing to do with entertainment. They'd be much more practical than you know, just giving us a bit of a thrill as we

play a game. But I thought that was really neat the idea of im acting your physical world through your virtual actions. See it's that that marriage or the physical and virtual that I find so fascinating. And again that's one of the reasons why I like augmented reality so much, is that idea that bringing those two worlds closer and closer together until we get to the point where you, you know, if you don't like how your day is going,

you just at control of the lead. But but the connect has also been used in lots of other applications, things like digital puppetry, where people have built a digital character that they could manipulate by doing motion controls. And some of them are very much like puppetry. Like you you have the digital version of a sock puppet. Someone holds up their hand and moves their hand and then the sock puppet reacts in a very similar way. Um, it's got no strings to hold it down, and it's

just virtual strings. Yeah, it's pretty interesting. And then I've even seen stuff where people have added in special effects on top of whatever their actions are in real time. So remember the connect also has a color camera in it, so it's not just this depth sensor. It's also a way of either streaming or even recording video depending on what you're doing and how you know what you have

the connect hooked up too. So let's say you're streaming video on a big screen and it's capturing whatever you're doing in front of the connect and you're holding a uh like a plane dowel, right, but you've created this program that overlays a an effect on top of that dowel. So now it looks like on the screen you're wielding a lightsaber from Star Wars. And as you as you move around, the video version of you is swinging a lightsaber, even though the real version of you swing a dowel.

On the screen, it's a lightsaber. You know. That's something that I that was one of the first applications I saw actually one of the first hacks. Yeah. Well it's uh not completely unlike what they actually did in the movie. Yeah, it's just all done in real time as opposed to

post production. That's what's amazing is that it's done right then, you know, because I've seen I've seen people who have created their own Star Wars type videos where they've gone in and they've used some pretty sophisticated post production video software to go in and paint an object right within the software and say, anytime you see this object, overlay this effect on top of it, which is really sophisticated, and it tends to be way faster than the old way.

But this is even faster. It's essentially instantaneous, which to me is phenomenal. So anyway, there are so many other hacks out there. If you just do a search online for Connect hacks, you are going to find blogs and articles and videos about amazing stuff people are doing with this basic piece of video game hardware. Yeah, if you if you've done any yourself, please let us know. We'd

love to hear about it. Yeah, definitely, Yeah, I've got if you've got any information about Connect hacks you've done, or any videos or whatever, sind some links our way, because you know, will we'll be glad to take a look at it, and who knows, maybe we'll be able to post them up on Facebook and create even a gallery of of things that people have done. And that wraps up our discussion about the Connect. And as I was saying earlier, since two thousand and twelve, the Connect

is pretty much gone by by. No one really talks about it anymore, no one's really developing for it anymore. And part of the reason for its failure was largely because there was a an apparent lack of compelling content for the Connect. There was initial interest in developing for it, and then it just kind of fell off. The challenges for developing for the platform We're probably too great for

the returns that were being seen. And there were a lot of games that kind of gave the Connect a bad name, games that gamers just didn't really and I hate to use this word connect with and so it

really just kind of floundered. It's also a shame that while it did very well in hacking circles for a while, Microsoft made some moves that made it harder for hackers to use the Connect, and they were creating all sorts of really cool implementations of the connect sensors and using them for things like machine vision, and uh, it was

really compelling stuff. So it's kind of a shame that that fell apart two, but it also shows that often the hacking community will take stuff that was intended for one purpose and then repurpose it for something else and make something really incredible out of it. If you guys have suggestions for future episodes of tech Stuff, send me an email the addresses tech Stuff at how stuff works dot com, or you can pop on over to our website,

that's tech stuff podcast dot com. There you're gonna find an archive of all of our episodes, including these classics. You will also find links to our presence on various social media platforms. You will find a link to our online store, where every purchase you make goes to help the show, and we greatly appreciate it, and I'll talk to you again really soon. Yeah. Tech Stuff is a

production of I Heart Radio's How Stuff Works. For more podcasts from I heart Radio, visit the i heart Radio app, Apple Podcasts, or wherever you listen to your favorite shows.