Rerun: Augmenting Your Reality

the Internet in an insecure way. But before we get to that, I thought maybe we could revisit this classic episode in which I talk about augmented reality. So this episode is titled Augmenting Your Reality, and it originally aired just a couple of years ago, and I thought it'd be fun to revisit it. So let's sit back and

enjoy this classic episode. So I thought I would do a deeper dive, a bigger explanation about what augmented reality is, what it's all about, how it works, and sort of the applications we might put a r toward things that you know, was it good for tons of stuff? As it turns out, So the first thing we should do is probably defined some terms, because if you haven't really looked into augmented reality and you aren't familiar with a R,

you might just be lost. I'm gonna define it all for you right now, because that's the kind of stand up guy I am. Technically speaking, augmented reality is using digital information to enhance or augment and experience in our physical real world. So the way we usually see this implemented involves some sort of display that has an image of the real world on it and it overlays digital

information on top of that itch. So think of like a camera's viewfinder, like an LCD screen on a camera, and it actually labels the buildings that are in view.

When you're out on the street and you hold the camera up or a smartphone or even a wearable device like a head mounted display that you can look through so you can see the real world, you're not just staring at a screen, or if you are staring at a screen, you're staring at a video feed that is provided by an external camera amounted just on the other side of the screen. So it's like you're looking through a display in the first place, but then on top

of that view you have this digital information. That's the most common implementation we talked about, but it's not the only one. Augmented reality does not have to only be or even involve visual information at all. You could have audio only augmented reality, for example. But the whole idea is that it's something that's created digitally to enhance your experience in the real world. Now we can contrast this

with the concept of virtual reality. Virtual reality, of course, is a term where you create an experience completely through computer generated means computer is making all the things you see and here, and maybe even beyond that if you have really sophisticated UH setups, so you might have some haptic feedback. Haptic refers to your sense of touch, So if you have haptic feedback, that means you're getting information

feedback through your sense of touch. Common example of this is a rumble pack inside a game controller, where you know you fire a gun and a first person shooter and your controller rumbles as a result, letting you know that you are in fact, unleashing virtual destruction upon all you survey. Well, the same thing can be true with virtual reality set up. So virtual reality is all about

constructing an artificial reality a simulated reality. Augmented reality is all about enhancing the one that we are actually in. And then there's also mixed reality. Mixed reality is kind of sort of in between the two. You might have some physical objects within a room that are also mapped to a virtual environment, and then you use something like

a head molded display to enter the virtual environment. That's what looks like you're inside, but you have physical objects in the room around you that are also mapped to the virtual world, Meaning you could pick up this physical object and you would see that reflected within the virtual world, where you might pick up a sword and shield or

move a chair or something along those lines. So augmented reality, virtual reality, and mixed reality are all kind of inter related, so much so that their histories also are very much inter related. And there's some people who try to elect these different technologies, these different approaches and put them under a common umbrella, and they tend to use the phrase alternate reality, which is unfortunate because that's also a r but alternate reality is kind of the umbrella for virtual,

augmented and mixed reality. Uh that that that kind of gives you the definition of those basic terms, and it is important to understand them because they're becoming more and more important today. You are already probably aware of a lot of VR headsets that are out there on the market as well as VR uh Well, they're they're kind of like cases that you slide your smartphone into, so your smartphone becomes the actual display on a VR headset. The headset itself is more or less just a head

mounted case for your phone. We've seen a lot of those come out over the last few years. We've also seen a lot of a R applications come out, typically for things iPads and smartphones, but we've also seen some hardware come out that for wearable devices that falls into the augmented reality category, stuff like Google Glass, which i'll talk about more a little bit later in this episode.

For augmented reality to work, to get this enhanced experience of reality around you, there are a lot of technological components that have to come together so that you actually do get an experience that is meaningful. You know, you you have to have technology that quote unquote knows where you are and what you are looking at or what you are close to in order to get that augmented experience.

It wouldn't do me any good. If I put on an augmented reality headset, for example, and stared at let's say a famous painting, and instead of getting information about the famous painting, I see an exploded view of an

car engine, that would make no sense. So you have to build in technologies in order for the a R to understand what it is you're trying to do and to augment that experience, which meant that we had to wait a pretty good long time for the various technologies that we used to to create this relationship to mature

to a point where it was possible. Sometimes we had technologies that would allow us to do it, but it required uh tethering headsets to very large computers, which meant that you didn't have really any mobility, and uh it really limited the usefulness of the actual application. In other cases, you could say things like your head tracking technology was absolutely necessary for a R to develop the way it did.

GPS technology as well. Remember it wasn't that long ago that we ordinary mere mortals didn't have access to really accurate GPS information. For a very long time, that was purposefully made less accurate. It was a matter of national defense. It wasn't until the nineties that you started to see GPS become more accurate for the basic consumer. Wait, back in the day, you might get accuracy of up to around a hundred meters, which is not great if you're

looking for the next place to make your turn. If it's a hundred meters away, that's that's pretty far. But now it's within a few feet, so it's much better. That sort of stuff all had to come together in order for augmented reality to become a viable I almost said a reality, but that just starts to sound redundant at any rate. Let's talk about some of these technologies

we are We really need things like gyroscopes, accelerometers. These help devices understand their orientation where they are in respect to something else, like are they for a smartphone it might be is it in landscape mode or portrait mode? But for a head mounted display, it would help give the the unit the information that needs to know which way you're looking, like are you looking to the east

or to the west. That kind of thing. UH also compasses obviously very important GPS sensors, image recognition software, but has become really important so that when you are looking at something UH, the system can actually identify what that is. In some cases you can get around this. You can design an a R system where let's say you make a movie poster and the a R application has the movie poster or animate in some way if you hold

up a smartphone that's running the appropriate app. So I'm just gonna take a movie from my past that does not have an a R movie poster or associated with it, but one that I can talk about as if it. We're a good example, and that has to be Big Trouble in Little China, universally declared the best movie that

has ever been made. So you've got your Big Trouble and Little China poster up on the wall, and you hold up your smartphone and you activate your Big Trouble and Little China movie marketing app, and the camera on your phone detects the poster it you know the posters there well. The app and the poster together are able to construct the augmented experience because there have been elements put into the poster that the app is looking for.

And once the app identifies that, like, it sees maybe eight different points on the poster, and because of the orientation of those points, it knows what angle it's at what height it's at in relation to the phone, and can give you on your display the augmented reality experience. In this case, it's obviously Jack Burton and the pork Chop Express eating a sandwich, because, as we know, the most riveting scene in the movie unfolds in this way.

So that would be kind of an augmented reality experience where you didn't have to worry about every possible application down in the real world. You made it for something very specific, which means in your software you can have the camera look quote unquote for these particular points of reference and thus create the augmented experience in that way.

If you want to take that and move it to the real world where you can see augmented information about just the world around you, it becomes way more complicated. You have to have very sophisticated image recognition software so that the camera picks up the images, the software processes the information, identifies what those images are, and gives you the relevant information. So working with all the sensors, augmented reality can make this a possibility. So another example, let's

say you're out on the street in Atlanta. You're here in my hometown at and a Georgia and you're looking at a building and you wonder what it is, and you hold up your phone and you've got your little map app that allows you to look at a real world setting and tells you information about it, and it tells you it's the Georgia Aquarium. Well, first of all, you would probably know that already because the signage there

is actually pretty good. But the point being that this would be something that would tap into the GPS cordints on your phone, so it would know where your location was and help narrow that down. The compass would tell it what direction you are facing the the camera angle. Also we have some image recognition going on there. The

accelerometer tells the orientation of the phone itself. All of this data together would give the software the information needed for it to display the label Georgia Aquarium on your phone. And it all happens in an instant that's pretty amazing. Typically, you also have to have some other method to communicate with a larger infrastructure, because we don't have the capability of building an enormously powerful computer that has all this real world information programmed into it and make it a

handheld or wearable device. So usually you have to pair these devices with some other larger infrastructure. Sometimes it's a double handshake. For example, with Google Glass, you would use Bluetooth to connect Google Glass to a smartphone. Then the smartphone would have the connection to the larger internet through

your smartphones UH cell service provider. So while you're experiencing the augmented reality through the Google Glass, it's actually communicating through your phone to the infrastructure to get the data it needs to show you the information. It's showing you very important elements. And all of these components, like I said, came together more or less around the same time. Most of them were being developed independently of each other, and

it's just that now we're seeing them all converge. That's an old favorite word here at tech stuff converge together to create the augmented reality experience and make it possible. So how did we get here? How did these different elements develop? Well, there are a whole bunch of technology pioneers who really create the foundation for augmented reality as well as virtual reality and mixed reality. But one that I think we really need to concentrate on at first

is Ivan Sutherland. Now. Sutherland was born in Hastings, Nebraska in nineteen thirty eight, and as a kid, he was fascinated with mathematics, particularly geometry, and also with engineering. He began to study and experiment with computers while he was in school, and this was at a time where personal computers weren't a thing. There were no personal computers at

this point. Computers were actually pretty rare, and they were huge, and in fact, they often would rely upon physical media formats like punch punch cards or paper tape to read a program him. So you didn't even have a disc or like certainly nothing like a USB thumb drive or anything like that. You you actually had to put physical media into the machine for it to read and then execute whatever program you had designed for that device. He went to college at what is now Carnegie Mellon University

on a full scholarship. He graduated with a Bachelor of Science degree. He would then go on to earn a master's degree at cal Tech and a PhD in electrical engineering from m I T And actually his doctoral thesis supervisor was Claude Shannon. And we talked about Claude Shannon back in the two thousand and fourteen episode Who Is Claude Shannon. Um. We recorded that not too long after Shannon's passing, So if you want to hear a really interesting story about a pioneer in computer science, you should

go check out that two thousand and fourteen episode. Back to Sutherland. For his thesis, he created something called sketch pack AD and that was really, by most accounts, the first computer graphical user interface, or gooey. A graphical user interface means that you interact with the computer through graphics representing various commands on the computer. Windows and the Mac operating system are both examples of graphical user interfaces, as

is the interface on your smartphone. If you have a smartphone where you choose applications on a screen, that's a graphical user interface. Well, Sutherland created what is largely considered to be the first one of those. After college, he entered military service and he was assigned to the National Security Agency. We have great friends there. I assume I'm sure they're listening, because they're listening to everything at any rate.

He entered the n s A as an electrical engineer, and in nineteen sixty four he replaced J. C. R. Lick Lighter as the head of DARPA's Information Processing Techniques Office or ip TO. And also by back then DARPA wasn't DARPA, it was just ARPA. Uh so this is the same group, by the way, that would end up doing a lot of work that would form the ARPANETTE a few years later, and the Arpanette was the predecessor

to the Internet in some ways. At least, the ARPANET was what ended up being the building blocks for the infrastructure that would become the Internet. Now, all of that work happened after Sutherland had already departed the organization. His work became a fundamental component of both virtual and augmented reality. As I mentioned earlier in n he wrote a piece and essay. It's very short, it's very easy read, and you can find it online. The title of the essay

is the Ultimate Display. And if you ever do any research and virtual reality or augmented reality, this essay is going to pop up in your research, so go ahead and read it. It's like two pages long, so it goes very quickly. In that essay he talked about several ideas, including the idealized display, the ultimate display, something that would

be the furthest you could go with display technology. Now, keep in mind in his time, by his time, he's still alive, by the way, but this time in the nineties sixties, he uh, you know, things were just restricted to monitors. You might have a light pen, but usually you would just use a keyboard. Like it was pretty bare bones. But he said, let's push this as far as we can imagine it. And in his example, he thought of a room that would be completely controlled by computers.

Everything you would experience within that room would be generated by a computer. Everything you see here, smell, taste, and touch, all of it generated by computers. The computer would even be able to form physical objects out of ror matter itself. Now, he wasn't suggesting that this would ever be a device that we would actually be able to build. He was just saying, what is the ultimate incarnation of display technology?

And if you read it, you realize, oh, this is where the Star Trek Next Generation writers got their idea for the Holidack. But unlike Star Trek the Next Generation, the Ultimate Display would not go on the fritz every other episode and try to kill the crew. It was better than that. The Ultimate Display was sort of a a foundational like Philosophically, it was foundational for virtual reality

and augmented reality. This idea of a very immersive experience where you, as a user are surrounded somehow by this computer generated experience. And that's true both with augmented reality and virtual reality and augmented reality. The real world is still there, but you get this enhanced experience that is completely computer generated. So in nineteen sixty eight, Sutherland and a student named Danny Cohen would create a VR, a R head mounted display or h MD, and they nicknamed

it the Sword of Damocles. Why because you had to suspend it from the ceiling. It was too heavy to wear on your head. You needed it to be nice and sturdy. Uh. It included transparent lenses, which meant you could overlay computer information on the lenses themselves, and thus you could look through the lenses at the real world and have these wire framed graphics on top of what

you were looking at. And it also had a magnetic tracking system, meaning that it had sensors that could detect magnetic fields, and as you turned your head or you change the inclination of your head, it would change the magnetic field and this would be relayed as a command to the visual center. The actual lenses themselves, so that it would the change would be reflected in what you saw.

So if you have a virtual environment and you turn your head to the left, you want the view within the virtual environment to go to the left too, But without head tracking technology that's impossible. So this was a very early example of head tracking technology. And again it used magnets magnetic fields in order to do that. Uh.

Obviously it's also really important for augmented reality. Again, if the A R system doesn't detect that you are looking around, then you're not getting relevant information, not for the specific thing you are looking at. Anyway, as I said, the graphics were pretty primitive. There were wire frame drawings, but they still showed that this was a viable approach to

technology using HMD for augmented or virtual reality use. Oh and one other note I should make so a lot of people say the sort of Damocles was the first head mounted display, and they say, you know, this is the first HMD was made in nineteen I take issue with that. I don't think of the sort of Damocles as the first head mounted display. That to me should go to a different invention called the head site h E A D s I g h T. Now that was developed by Phil Coo, and unlike the sort of Damocles,

it didn't create a virtual world. Instead, the head site was sort of a remote viewfinder for a video camera. So imagine that you've got a camera mounted on a mechanical, uh swiveling mount, so you can move it left right, you can change the orientation the inclination as well, and then you have that mapped to a head mounted display, so that if I put the display on and I looked to the left, the camera pans to the left. If I to the right, it pans to the right.

That sort of thing. It was meant to be a way for for people to operate a camera in a remote location that might not be very friendly to a human being standing there. For example, the exterior of an aircraft. You could have a camera mounted on the outside of your aircraft that would allow an engineer on the inside to look around and maybe help a pilot land or navigate in a dangerous situation, or just get an idea

of the status of the aircraft itself. This was very much a technology that was being pushed by the military, an idea to create more military uses using this technology. To make the military more competent, more adept at very rapid changing situations on the technology front. So headsite proceeded the sort of Damocles by about seven years. That came out around night sixty one. But again it wasn't a virtual reality headset or an augmented reality headset. It was

kind of a like I said, a remote viewfinder. But still I consider that to be the earliest head mounted display, not the sort of Damocles. However, Sutherland would end up going on to make lots of other contributions in computer graphics as well as the overall concepts that would guide both virtual reality and augmented reality development over the next

several decades. But now it'll be time for me to kind of move away from Sutherland to talk about some other developments that were important in a R. And before I get to that, let's take a quick break to thank our sponsor. All Right, we just left off with Ivan Sutherland. Now let's talk about a different father of augmented reality, Myra Krueger or Doctor Myron Krueger. In nineteen seventy four, Dr Krueger created an augmented reality lab called

video Place. Uh, he was really into this idea of seeing the interaction of technology and people in artistic ways. He really wanted to explore artistic expressions using technology and people working together. So he wanted to create an artificial reality environment that didn't require the user to wear special equipment. You wouldn't have to put on a head mold display, or wear special gloves, or use any kind of device to control your actions, because that's a barrier between you

and the experience. Instead, his version consisted of a laboratory that had several rooms all network together, and each room had a video camera in it and a projector and a screen. Now, the video camera would pick up the motions of the person inside the room, it would send information to the projector, which would then project the person's

silhouette on the screen. And the silhouette was typically a really bright color, and you could move around and your silhouette would move around, so you almost became like a puppet master controlling your own silhouette. But then he started to incorporate other things, like other elements that were virtually on the screen. The projector was projecting things that were on the screen but not in the actual real room itself. So imagine a ball and a ball is being projected

on the screen. Well, you could move around so that your silhouette would interact with the ball and the ball would bounce away. That sort of thing. So you would be able to interact with virtual environments by moving around in a real physical space. And while those objects weren't really there in front of you, you could see the representation of them on the screen. And this was really

powerful stuff. And remember I said these rooms were all network together, so you could actually have a system where a person in one room and a person in another room both have their silhouettes projected together in their respective rooms on the screen, and your silhouette would be one color, the other person's silhouette would be a different color, and

you could interact with one another. And according to reports from this art experiment, they noticed that whenever people would have their silhouettes cross one another, they would actually recoil in their physical rooms. Keep in mind, they're in different rooms, they're not in the same one together, they would recoil as if they had made physical contact or bumped into someone. So it showed that there was a very powerful psychological

element to this virtual presence. And again that psychological element plays a huge important role in VR and a R research and development, not just for creating products, but just to understand how we process information and incorporated into our sense of reality. Not to get too deep for you guys, So experimentation in the field continued over the years. In the early nine eighties, Dr Krueger would write a book

and publish it about artificial realities. But while the principles for augmented reality were established, the technologies were still rather unwieldy. They were large, they weren't reliable, and it would require several years of work to improve those technologies, to create miniaturization strategies to get the elements down to a size that was more practical for that sort of use and wouldn't require you to have a head mound display mounted

to the ceiling. And all of that took time, but you could tell that the ideas underlying augmented and virtual reality were already in place. There was a Boeing researcher named Tom Coudell who coined the term augmented reality, and he was specifically using it to talk about this approach to overlaying digital information on top of our physical world

to enhance it in some way. Now Dr Cardell earned a PhD in physics and astronomy from the University of Arizona, and before contributing the term augmented reality to the public lexicon, he did extensive work and artificial intelligence research and development. He also became a professor in the fields of electrical

and computer engineering at the University of New Mexico. So when he was working with Boeing, he used this phrase to talk about specific system he was working on, an augmented reality system, and the whole purpose of this was to help people who were helping construct airplanes lay cables properly.

The whole idea was to use this system so that uh, an electrician can see exactly where the cable needed to go inside the partly constructed cabin of an aircraft aft, and that way you could follow the directions that you see through your display, lay the actual cable down where the guide tells you to go, and then you would have a properly wired airplane. Uh. And I'm sure, as we're all aware, properly wired airplanes are good airplanes. Improperly

wired airplanes are not so good. So it was a very important system to make this much more smooth and fast and it meant that you didn't have to have as as many experts to guide the process. You could actually have someone come in who had never done this before and just follow the directions through this augmented reality set system and they could wire the airplane properly. So

really clever means of using augmented reality. Also, we would end up seeing that same sort of philosophy used again and again in the future in more sophisticated UH types of technology, but it was the exact same approach, exact same idea underlying it. In Lewis Rosenberg proposed a system that the Air Force could use to allow someone to control devices from a remote location, and that consisted of a video camera which would provide the visual data to

the user through a head mounted display. They would wear the display on their heads or they would look at a screen, but typically they'd wear a display, and then they would also wear an exoskeleton on their upper body that would allow them to control some sort of robotic device,

typically robotic arms. And usually the way this would work is that the display was designed in such a way with the video camera so that the view that the person had it made it look like the robot arms were their actual arms, which required a little bit of trickery on the part of Rosenberg. They had to fudge the distances between the video camera and the robotic arms to give this this sort of feeling that the robot

arms represented your actual arms. So you move your arms inside the exoskeleton and the robot arms would move as well at their remote location, so you it's kind of like a really fancy remote control. Now imagine that the robot arms are holding various tools. Uh. The suit would also provide haptic feedback, that touch based feedback to let a user know more about what is going on when

they're operating the arms. So if you were to have do something that would make a robot arm encounter resistance, then you would feel haptic feedback in the suit that would indicate, oh, you're you're going beyond the parameters of where this robot arm is capable of going. So you learned very quickly where where you can operate within that suit and make sure that you are not pushing it beyond its limits. You could also uh end up using

these tools to do various things in this remote environment. Now, Rosenberg called his system virtual fixtures, which meant that the user would see these virtual overlays on top of a real environment that they were looking at. So I'm going to give a very basic example that to illustrate this, because it's hard to imagine, it's hard to get it across in words. But let's say you're looking through a headmund display and in front of you is a board,

wooden board, and it's just a regular wooden board. There's nothing painted on it or anything in the real world, and it's in a room that's across the building from you. You cannot see this with your own eyes. You can only see it through the video camera. The virtual fixture overlay might be a series of circles, and the circles are things that you were meant to cut out of the board using the robot arms and a tool that's right there in side the physical environment, across the building

from you. So you follow the patterns that you see in this virtual overlay and you complete the task. That's a very simple example, and uh, this system was meant to allow for that. That's what he would call the virtual fixtures, these overlays that you would see that would appear to be real, but actually we're not present in

the physical environment itself. Now, also in a group of researchers at Columbia University, we're proposing a system that they called the knowledge based Augmented Reality for Maintenance Assistance a k a. Karma Cute. Their approach was pretty novel. They pointed out that while augmented reality had tremendous potential, it also had a really big barrier and that it takes an enormous amount of time to design or animate and

implement these graphic overlays for a R applications. So say you're in a room and you're looking at different objects, and little labels are popping up for each object. If you're having to do all that by hand, it takes a huge amount of time. What they wanted to do was create artificial intelligence systems, or at least techniques to

generate graphics automatically on the fly. So this would be similar to using image recognition software, so that if you look at a specific box, let's say, the image recognition software might be able to map that box to a specific product and thus give you an overlay of information about the product that would be inside that box, and it would be able to do all this automatically. It would not require a human programmer to go through and and look at every single product in every single type

of box and program all that out. That would be ridiculous, It would take forever. So it was the work of this group with Karma that really started the ball rolling with this AI approach to automatically fill in that information and make a are a more practical experience. Around the same time, between ninety two and nine, the Laurel Western Development Labs, which was a defense contractor, began to work with the U. S Military to create a R systems

for military vehicles. And you can understand very quickly how a R would have enormous potential for military applications. And in fact, a R is very commonly used in lots of different things like pilot helmets where it helps pilots keep track of targets UH and identify a potential threats that kind of thing. But in this case they were really looking at creating a augmented reality system that would create virtual opponents for people working in simulated wartime conditions.

So really a training program. Imagine that you're operating an actual military vehicle like a tank, and you have a view outside that is really an augmented reality system, so you're actually looking at the real world around you. You aren't just sitting in a simulator inside of building. You are out there in the field controlling a real vehicle

moving around in real terrain. But you also see virtual representations of enemies in that real terrain, and you can practice maneuvers and firing on enemies that sort of thing, probably not using live ammunition at that point, but having a more realistic simulation in a real environment, so that you're not just trying to create a totally virtual scenario. Anyway, that work was done in the world, wouldn't really learn about it at large until about nine because that's the

way the military works. They're not so eager to talk about their stuff while are still doing it. Meanwhile, at the same time, artists were continuing to explore the relationships between physical performers and virtual LM. And so you remember I talked about Dr Krueger earlier. While in nine different artist Julie Martin would create a piece called Dancing in Cyberspace, and in that piece, dancers on a physical space or

a physical stage. We're able to manipulate virtual objects, so an audience would be able to see both the physical performance by the dancers and the virtual reactions the things that happened within the virtual environment as a result of the dancers moving around their physical space. Pretty neat. Two researchers, Recky Motto and Nagao created there the first real handheld a R display, but it was a tethered display. It

wasn't free form. You couldn't just take it anywhere. It was called Navy Cam, and you had to have a tether like cable essentially connect the Navy Cam to a workstation. But it had a forward facing camera and you can use a video feed to go through this and held device through the cable to the workstation, and it could detect color coded markers in the camera image and display information on a video see through view. So you can

get that augmented reality experience. Obviously very limited, you know, you could not just carry this around with you everywhere you go, but it showed the ideas behind augmented reality could in fact be realized in a handheld format. Now it's just a matter of getting those different components small enough to all fit in a self contained mobile form factor. Now, in the late nineties, we started seeing televised sporting events

featuring augmented reality elements, or at least you did. I don't watch sports ball, that's not entirely true, but I don't watch football or hockey, American football or hockey, and both of those were the sports that really got them. First off, I'm gonna backtrack. I used to watch hockey, but then Winnipeg stole the Atlanta Thrash years from me. Winnipeg, Okay, getting back to hockey. So hockey had the Fox track system, which Fox put into hockey games so that you could

easily follow the puck. Instead of trying to watch this little bitty black disc spinning around, You've got to watch this very bright, highlighted neon colored disc that everyone hated. And after about two seasons, Fox stop doing it, and people were happy. Adila Thrashers moved away, and then it was just miserable. American football would follow suit in the late nineties and have the first down line introduced, where they could on live video overlay the first down line.

Usually it's a bright yellow line that indicates how far the offensive team needs to go. And by offensive, I mean they're on the offensive. I don't mean they offend my sensibilities. I'm not that against American football, but it showed how far they would need to go in order to estab a first down, which I am told is something you want to do. Uh. That would start to get employed in and over time we would see that increase where eventually Skycam was able to even use this system.

At first it wasn't. You could get a Skycamp view, but you couldn't do the overlay of the first intent line until later. Well, I've got a lot more to say about augmented reality, but before I do, let's take another quick break to thank our sponsor. Okay, we're back. Let's skip ahead to I guess it's not really skipping

I just talked about. Let's plot ahead to That's when NASA's X spacecraft was using an a R system as part of its navigational tools, so people back on Earth could look at a view from the spacecraft a camera mounted on the spacecraft, and on top of that view they could overlay map data to help with navigation. And all of that, of course was controlled back here on Earth. But it was sort of an experiment to see how augmented reality could be incorporated into space exploration missions. In

the future and make them more effective. Also, in the Navy began work on the Battlefield Augmented Reality System or BARS, which is a wearable a R system for soldiers. You've probably seen various implementations of this over the years. Is obviously evolved since nine It's one of those pieces of technology that some soldiers took to, but a lot just

felt that it created unnecessary distractions. Technology and warfare is very very difficult because there are sometimes where we think, oh, more information and is always better, but in some cases that doesn't seem to hold true. And for some people with these head mounted displays or or really it's heads up displays HUDs uh, that can sometimes be the case, depends on the implementation. In two thousand, hiro Katsu Kato

created a software library called a R Toolkit. Very important software library was also open source, so anyone could contribute to it, modify it. Brent put out a new version that sort of stuff, and it uses video tracking to overlay computer graphics on a video camera feed and it's still a component for a lot of a our experiences today.

Later on in the two thousand's this would be adapted so that it could also be used in Web experiences, not just native experiences to specific devices, and we continue to see are built into new experiences, including smartphone and tablets. By two thousand four, some researchers in Germany were creating a r apps that could take advantage of a smartphones camera.

But two thousand four's pretty early for smartphones. It really would would be a few years before this would truly take off, because that's when Apple came out with the iPhone in two thousand seven. That was the real revolution in smartphone technology. There have been smartphones before the iPhone, don't get me wrong, and many of them were really good, but the iPhone was something that caught the public's attention

and made smartphones sexy. And because of that, there was a ton of money poured into the smartphone industry as well as not just Apple, but also to other companies, like the companies that were offering Android smartphones. But I think we can really thank Apple for all of that happening in the first place, especially things like seeing that accelerometer where you could switch from portrait to landscape mode.

I remember everyone freaking out about that when Steve Jobs showed it off in two thousand seven at Macworld and everyone thought, Wow, this is the this is amazing. Well, we take it for granted now, but it was a

big deal then. So once that smartphone revolution happened, it was a landslide victory for both augmented reality and virtual reality research and development because it meant that so much money was being poured into creating newer, thinner, more capable smartphones that we saw an explosion in technological development that could also be used for virtual and augmented reality experiences. So, for example, I think of those sensors I talked about earlier,

accelerometers and gyroscopes, that sort of thing. Well, we saw a lot of development in those spaces in order to make smartphones better, and people who are working in a R and VR experiences can take advantage of those same sensors either creating apps specifically four smartphones. Thus you don't have to build any other hardware, you just use existing hardware, but that limits how you can use it, right because you don't typically wear your smartphone directly in front of

your face. Or they could end up taking advantage of those new, smaller sensors and incorporate them directly into brand new hardware, like various types of wearables like Google Glass, for example, but that would be a few more years. In two thousand eleven, Nintendo launched the Nintendo three D S,

which included a camera. It was, you know, the three D capable handheld device, and included actually a pair of forward facing cameras, so you could take three D photos if you wanted to, and it also had some a R software included with it. You would get these special Nintendo cards kind of like playing cards, and if you were to point the camera of the three D S at the card and look at the screen, you would see a little virtual three dimensional character pop up on

the card. So Mario would be an obvious example. You put the Mario card down on the table, you hold up the three D S, and you aim the camera at the card, and you look at the screen and there's Mario, and Mario appears to be jumping around on your physical table. Now, obviously, if you look off of the display, there's no Mario jumping around, but on the

display there he is, and it was pretty cute. I remember being really impressed with this very simple implementation of a R when we got our three DS, and then I took our three DS apart and then I took pictures of it, and then I posted on Twitter and people got sad, it's a great day. In two thousand thirteen, Google introduced Google Glass. That was the wearable that included a small display position just above the right eye. Uh so when you look straightforward, you could tell that there

was something kind of above your natural eyeline. But it didn't get in the way too much. You too to look at the screen. You actually had a glimpse, you know, the glance upward, and then you could see what was on the display. Google Glass had augmented reality features like crazy. You could see video calls. You could actually use the glasses to not just to take a video call, but show the other person what you are looking at so

they could see from your point of view. You could also overlay directions, so if you're walking down street, you could glance up at the screen and it would tell you if you need to keep going straight or turn laughter, turn right, that kind of thing. It was really useful. Um I had a pair of these Google Glass and

I really liked the direction they were going in. I felt that it wasn't a fully realized product at the time, and eventually Google agreed and after a couple of years, they took Google Glass off the market and early and now you can't get them anymore. Uh. They were clever, but they were expensive, and they had some limitations. And like I was saying earlier, you know, it's hard to

build all the components you need into one headset. So Google Glass would communicate via Bluetooth to your smartphone, and your smartphone would act as the actual nexus point to the Internet. But it was a neat idea, uh, and I enjoyed getting to use them while I did, so I keep hoping to see a return of that kind of technology, but perhaps in a more mature and less expensive format. Now we've also seen applications similar to the ones we mentioned earlier, the ones that are meant to

guide people into laying out or repairing a system. We've seen that in the car world. Not too long ago, there was the MARTA system introduced by Volkswagen. MARTA makes me chuckle because that's also the name of Atlanta's public transportation system, but in this case, it stands for Mobile Augmented Reality Technical Assistance and it's specifically designed for mechanics

who are working on the XL one vehicle. So if you hold up an iPad that has this app on it, and the camera has pointed at an XCEL one and you look at the display, you'll see information overlaid on top of the car, including labels for all the different parts. So let's say you're a mechanic and you have to do a specific repair on this vehicle. You hold up the iPad, you look through the display, and you see

exactly what you need to do. It gives you a set of instructions that shows you how you need to do. It tells you where you need to stand based upon the angle of the view. So if you hold it up and it says no, you need to move about a foot to the right, you can do that. Then hold up the iPad again. I'll say, all right, you're in the right spot. Make sure you loosen this particular bolt first. That kind of thing, and it's meant to be an interactive maintenance guy in a way, maintenance and

repair guide. This is one of those applications of augmented reality. I think is a no brainer to me. It's a killer app The idea of having an ability to work with something you are not familiar with, but you're able to leverage the expertise of people who either designed it or built it, or just fully understand it, and get guidance based on their expertise in real time, so you're not having to go and consult a an article about

it or watch a YouTube video. You get step by step instructions overlaid on top of your view of that thing. To me, that's the most compelling use of augmented reality from a practical standpoint. There are a lot of other uses that I'll talk about towards the end that I think are also really super cool. So don't get me wrong, It's not the only one. But let's move on to thousand fifteen. That was when Microsoft would unveil the hollow lens, something I still want to try out. I have not

had a chance to try a hollow lens yet. That is a headset cable of advanced a R applications everything from what I was just talking about, giving you guidance, step by step instructions on how to do like a repair job on say an electrical outlet. You can even use a Skype system to call an expert who can then view your point of view and interact with that point of view. So Let's say I'm looking at the outlet.

The expert electrician i'm talking to can see what I see, and he or she can also make notes on the display which shows up in my field of view. So he or she might circle a specific wire and say you need to you need to remove that one first, and I know I need to do that one first because I can see which one they are talking about.

Or they might circle another wire and say, no matter why you do, don't cut this wire where the toilet upstairs will explode like lethal weapon too, and I won't do that because you know that guys like three days from retirement. So I have a heart. But no, this is this is a really neat idea, having this interactive ability to overlay the information from the world, the digital world, onto your physical world and beyond that. The hollowlens has

lots of other functions. It's not just something to do, you know, home repairs around the house. You can also use it for entertainment purposes, like you could create a screen that can show you video from various sources and you can assign it a place on a wall in your environment. Let's say that you're in your living room and you just create a screen so you can watch Netflix, and you slap it on a wall and it will stay in that same position relative to your point of view.

So if you look to the left or right, the screen stays where you put it as if it were physically there on your wall. But keep in mind it's just a virtual screen, and when you look back to that part of your wall, you'll see the virtual screen there playing whatever it was that you wanted to watch.

I think that's a super cool idea. And they've also shown off games like a game of Minecraft that uses hollow lens so you can actually view a Minecraft world sitting appearing to sit at any rate on top of a table, so you can walk around the table and view the Minecraft world from various angles and play that way. I think that's super neat. Don't know how compelling it is, because again I haven't tried it myself, but I really

like the idea. This year two thousand sixteen, are got another big boost from a little game called Pokemon Go. Although I have to admit this was a really primitive, base sick implementation of augmented reality. Really, it was not much more than just a In fact, it was nothing more than just an animated overlay that would exist on top of the camera view of your of your device.

So I'd say, I'm holding up my smartphone and I'm trying to catch a Jiggli Puff, and the Jiggli Puff is currently bouncing up and down on the sidewalk in front of me. That's about as far as the augmented reality actual experience would go. So very primitive. But because Pokemon Go became so popular so quickly, it really pushed the concept of a R to the front of the minds of people everywhere, including business owners who immediately said,

we need an augmented reality app. Whether they actually needed one or not is beside the point. A lot of people got into a R because of Pokemon Go. Uh, for both good and bad. I always think that you have to come up with the experience first. You have to understand why you need to use a specific strategy to create a specific experience and then build it. Not Hey, we need augmented reality makes something that's a R. To me, that's the backwards way of going about it. But what

do I know. I'm not not a programmer, so I'm sure the programmers feel in a similar way to me. But that's just a guess. Now. The future of a R depends heavily upon the applications we see, in which ones end up being successful, in which ones aren't. Right now, I would say that the best bet is to see more a R features built into smartphones and tablets. Uh not, maybe not necessarily built into them, but have apps available that create a OUR experiences for very specific contexts, Like

let's say it's a museum app. You might download a museum app on your phone, and when you go to the museum and you use your phone, you can get more information about the paintings and sculptures and or installations that you see in the museum. That's an easy one to understand. But that same app isn't going to be useful once you leave the museum, you no longer have

the context that it is tied to. I think that smartphones are probably going to be where the greatest development is going to be in the near term, because wearables is still really hard to do. We still don't have a consumer version of the hollow lens out available for anyone to purchase, and it may never come out as a consumer product. Microsoft hasn't shown a whole lot of

interest in making it a consumer product. Maybe that will change, but at the moment, I wouldn't hold my breath, so I would argue smartphones and tablets are pretty much where it's at. Maybe some implementation with some existing VR headsets which have external cameras mounted on them as well, like forward facing cameras, you could build a our experiences there.

Then it gets a little weird because you're you're also you know, you're looking at a monitor, so you're looking at a video feed of your roundings, and on top of the video feed you get the overlay. Same thing is true for your smartphones and tablets, by the way, but different that from the Google Glass implementation, where you're looking at the actual physical world, not a video representation

of it, but the real world. And then because the display itself that you are looking through is transparent, you're looking at a transparent overlay of digital information that gives you more info about the world you are in. I think a R is super cool. I think it's really got a lot of potential to change the world around us and to change the way we interact with the

world around us. You could imagine a dystopian future implementation of a r where we all have to wear glasses and we're constantly getting personalized commercials beamed at us whenever we look at anything. Like imagine walking past a store casually looking in the window and then getting a whole bunch of ads for all the stuff that's in the

store window. That would be obnoxious, and it's easy to understand how people would not want that, yet also easy to understand how that could possibly become a future or think of the future where your privacy is no longer even relevant, and you walk down the street and you look at all the people's faces who are also walking down the street, and you're getting names of everybody and

what they like and what they dislike. You know, what music they tend to listen to, maybe what they're listening to right now, And it's all because we've got facial recognition technology. Almost everyone has some sort of social media presence, so you can map that face to any public profiles,

try and find a match. If you found a match, you could bring back information to the person wearing the glasses, so I can look at somebody and say, oh, this, uh, this cute kid over here, she's got she likes punk rock music. I'm gonna I'm gonna impress her with my knowledge of the cramps. That probably wouldn't work. But the point being, this is pretty creepy and invasive, and so there are some negative implementations of a ARE that we

have to watch out for. Unless we get to a point where we just don't care about privacy at all anymore. Some would argue we're already there, and in that case, this implementation of a R may not sound creepy at all. It might just sound kind of cool, kind of the equivalent of walking into a store, seeing a person with a name tag and addressing them by name. If they don't remember they have a name tag on, they have

this moment where they think, do I know you? But if we're in a world where everyone can see everyone's name all the time, then well, for one thing, I won't ever have to worry about coming at a loss when I have to introduce my wife to someone. So that's a that's a plus side. And I hope you guys enjoyed that classic episode of text stuff. If you have any comments or questions, reach out on social media. We are available at text Stuff h s W both on Twitter and on Facebook. You can also go to

our podcast page that's tech Stuff podcast dot com. You can find a link to every episode we've ever recorded. There. You'll also find a link to our online store, where there's lots of tech Stuff merchandise, so you can go ahead and spend all that hard earned cash on really cool tech stuff products. I mean, seriously, we do have some really neat ones. If you haven't checked it out, go at least take a look at the designs. I think you'll find them amusing, and who knows, maybe you'll

find a new favorite tote bag or t shirt. Then remember, every purchase you make goes to help the show, and we greatly appreciate it, and I will talk to you again really soon. Tex Stuff is a production of I Heart Radio's How Stuff Works. For more podcasts from my heart Radio, visit the i heart Radio app, Apple Podcasts, or wherever you listen to your favorite shows.