Augmenting Your Reality

That's when Chris Palette was my co host and together we recorded an episode that we called What in the World Is Augmented Reality? And that was a pretty short overview of the concept, like twenty minutes long, and that included two different listener males. So I thought I would do a deeper dive, a bigger explanation about what augment the reality is, what it's all about, how it works, and sort of the applications we might put a R toward things that, you know, was it good for tons

of stuff? As it turns out, So the first thing we should do, it's probably defined some terms, because if you haven't really looked into augmented reality and you aren't familiar with a R, you might just be lost. I'm gonna define it all for you right now, because that's the kind of stand up guy I am. Technically speaking, augmented reality is using digital information to enhance or augment

and experience in our physical real world. So the way we usually see this implemented involves some sort of display that has an image of the real world on it, and it overlays digital information on top of that image. So think of like a camera's viewfinder, like an LCD screen on a camera, and it actually labels the buildings

that are in view. When you're out on the street and you hold the camera up, or a smartphone or even a wearable device like a head mounted display that you can look through so you can see the real world. You're not just staring at a screen, or if you are staring at a screen, you're staring at a video feed that is provided by an external camera amounted just

on the other side of the screen. So it's like you're looking through a display in the first place, but then on top of that view you have this digital information. That's the most common implementation we talk about, but it's not the only one. Augmented reality does not have to only be or even involve visual information at all. You could have audio only augmented reality, for example, But the whole idea is that it's something that has created digitally

to enhance your experience in the real world. Now we can contrast this with the concept of virtual reality. Virtual reality, of course, is a term where you create an experience completely through computer generated means a computer is making all the things you see and here, and maybe even beyond that if you have really sophisticated UH setups, so you might have some haptic feedback. Haptic refers to your sense of touch, so if you have haptic feedback, that means

you're getting information feedback through your sense of touch. Common example of this is a rumble pack inside a game controller, where you know you fire a gun and a first person shooter and your controller rumbles as a result, letting you know that you are in fact, unleashing virtual destruction upon all you survey. Well, the same thing can be true with virtual reality set up. So virtual reality is

all about constructing an artificial reality a simulated reality. Augmented reality is all about enhance and seeing the one that we are actually in. And then there's also mixed reality. Mixed reality is kind of sort of in between the two. You might have some physical objects within a room that are also mapped to a virtual environment, and then you use something like a head molded display to enter the

virtual environment. That's what looks like you're inside. But you have physical objects in the room around you that are also mapped to the virtual world, Meaning you could pick up this physical object and you would see that reflected within the virtual world, where you might pick up a sword and shield or move a chair or something along those lines. So augmented reality, virtual reality, and mixed reality are all kind of inter related, so much so that

their histories also are very much inter related. And there's some people who try to collect these different technologies, these different approaches and put them under a co an umbrella, and they tend to use the phrase alternate reality, which is unfortunate because that's also a r but alternate reality is kind of the umbrella for virtual, augmented and mixed reality. Uh that that that kind of gives you the definition of those basic terms, and it is important to understand

them because they're becoming more and more important today. You are already probably aware of a lot of VR headsets that are out there on the market as well as VR. Uh Well, they're they're kind of like cases that you slide your smartphone into, so your smartphone becomes the actual display on a VR headset. The headset itself is more or less just a head mounted case for your phone. We've seen a lot of those come out over the

last few years. We've also seen a lot of a R applications come out, typically for things like iPads and smartphones, but we've also seen some hardware come out that for wearable devices that falls into the augmented reality category, stuff like Google Glass, which I'll talk about more a little

bit later in this episode. For augmented reality to work to get this enhanced experience of reality around you, there are a lot of technological components that have to come together so that you actually do get experience that is meaningful. You know, you have to have technology that quote unquote knows where you are and what you are looking at or what you are close to in order to get

that augmented experience. It wouldn't do me any good if I put on an augmented reality headset, for example, and stared at let's say, a famous painting, and instead of getting information about the famous painting I see an exploded view of an car engine That would make no sense.

So you have to build in technologies in order for the a R to understand what it is you're trying to do and to augment that experience, which meant that we had to wait a pretty good long time for the various technologies that we used to to create this relationship to mature to a point where it was possible.

Sometimes we had technologies that would allow us to do it, but it required uh tethering headsets to very large computers, which meant that you didn't have really any mobility, and uh it really limited the usefulness of the actual application. In other cases, you could say things like your head tracking technology was absolutely necessary for a R to develop the way it did. GPS technology as well. Remember it wasn't that long ago that we ordinary mere mortals didn't

have access to really accurate GPS information. For a very long time that was purposely made less accurate. It was a matter of national defense. It wasn't until the nineties that you started to ce GPS become more accurate for the basic consumer. Wait, back in the day, you might get accuracy of up to around a hundred meters, which is not great if you're looking for the next place to make your turn. If it's a hundred meters away, that's that's pretty far. But now it's within a few feet,

so it's much better. That sort of stuff all had to come together in order for augmented reality to become a viable I almost said a reality, but that just starts to sound redundant at any rate. Let's talk about some of these technologies we were. We really need things like gyroscopes, accelerometers. These help devices understand their orientation where they are in respect to something else, like are they For a smartphone, it might be is it in landscape

mode or portrait mode? But for a head mounted display, it would help give the the unit the information it needs to know which way you're looking, like are you looking to the east or to the west. That kind of thing. UH also compasses obviously very important GPS sensors, image recognition software, but has become really important so that when you are looking at something UH, the system can actually identify what that is. In some cases you can

get around this. You can design an a R system where let's say you make a movie poster and the a R application has the movie poster animate in some way if you hold up a smartphone that's running the appropriate app. So I'm just gonna take a movie from my past that does not have an a R movie poster associated with it, but one that I can talk about as if it were a good example, and that has to be Big Trouble in Little China, universally cleared

the best movie that has ever been made. So you've got your Big Trouble and Little China poster up on the wall, and you hold up your smartphone and you activate your Big Trouble and Little China movie marketing app, and the camera on your phone detect the poster. It's you know, the posters there. Well. The app and the poster together are able to construct the augmented experience because there have been elements put into the poster that the

app is looking for. And once the app identifies that, like, it sees maybe eight different points on the poster, and because of the orientation of those points, it knows what angle it's at, what height it's at in relation to the phone, and can give you on your display the augmented reality experience. In this case, it's obviously Jack Burton and the Pork Chop Express eating a sandwich, because, as we know, the most riveting scene in the movie unfolds

in this way. So that would be kind of an augmented reality experience where you didn't have to worry about every possible application out in the real world. You made it for something very specific, which means in your software you can have the camera look quote unquote for these particular points of reference and thus create the augmented experience

in that way. If you want to take that and move it to the real world where you can see augmented information about just the world around you, it becomes way more complicated. You have to have very sophisticated image recognition software so that the camera picks up the images, the software processes the information, identifies what those images are, and gives you the relevant information. So working with all

the sensors, augmented reality can make this a possibility. So another example, let's say you're out on the street in Atlanta. You're here in my hometown, Atlanta, Georgia, and you're looking at a building and you wonder what it is, and you hold up your phone and you've got your little map app that allows you to look at a real world setting and tells you information about it, and it

tells you it's the Georgia Aquarium. Well, first of all, you would probably know that already because the signage there is actually pretty good. But the point being that this would be uh something that would tap into the GPS coordinates on your phone, so it would know where your location was and help narrow that down. The compass would tell it what direction you are facing the camera angle. Also, when you have some image recognition going on there, the

accelerometer tells the orientation of the phone itself. All of this data together would give the software the information needed for it to display the label Georgia Aquarium on your phone. And it all happens in an instant that's pretty amazing. Typically you also have to have some other method to communicate with a larger infrastructure, because we don't have the capability of building an enormous, sleep powerful computer that has all this real world information programmed into it and make

it a handheld or wearable device. So usually you have to pair these devices with some other larger infrastructure. Sometimes it's a double handshake. For example, with Google Glass, you would use Bluetooth to connect Google Glass to a smartphone. Then the smartphone would have the connection to the larger

internet through your smartphones UH cell service provider. So while you're experiencing the augmented reality through the Google Glass, it's actually communicating through your phone to the infrastructure to get the data it needs to show you the information. It's showing you very important elements. And all of these components, like I said, came together more or less around the same time. Most of them were being developed independently of each other, and it's just that now we're seeing them

all converge. That's an old favorite word here at tech stuff converge together to create the augmented reality experience and make it possible. So how did we get here? How did these different elements develop? Well, there are a whole bunch of technology pioneers who really created the foundation for augmented reality as well as virtual reality and mixed reality. But one that I think we really need to concentrate

on at first is Ivan Sutherland. Now. Sutherland was born in Hastings, Nebraska in nineteen thirty eight, and as a kid he was fascinated with mathematics, particularly geometry, and also with engineering. He began to study and experiment with computers while he was in school, and this was at a time where personal computers weren't a thing. There were no

personal computers at this point. Computers were actually pretty rare and they were huge, and in fact, they often would rely upon physical media formats like punch punch cards or paper tape to read a program. So you didn't even have a disk or like certainly nothing like a US Beat thumb drive or anything like that. You you actually had to put physical media into the machine for it to read and then execute whatever program you had designed for that device. He went to college at what is

now Carnegie Mellon University on a full scholarship. He graduated with a Bachelor of Science degree. He would then go on to earn a master's degree at cal Tech and a PhD in electrical engineering from m I T. And actually his doctoral thesis supervisor was Claude Shannon, And we talked about Claude Shannon back in the two thousand and fourteen episode who is Claude Shannon? Um. We recorded that

not too long after Shannon's passing. So if you want to hear a really interesting story about a pioneer in computer science, you should go check out that two thousand and fourteen episode. Back to Sutherland. For his thesis, he created something called sketch Pad and that was really, by most accounts, the first computer graphical user interface or gooey. A graphic goal user interface means that you interact with the computer through graphics representing various commands on the computer.

Windows and the Mac operating system are both examples of graphical user interfaces, as is the interface on your smartphone. If you have a smartphone where you choose applications on a screen, that's a graphical user interface. Well, Sutherland created what is largely considered to be the first one of those. After college, he entered military service and he was assigned to the National Security Agency. We have great friends there. I assume I'm sure they're listening, because they're listening to

everything at any rate. He entered the n s A as an electrical engineer, and in nineteen sixty four he replaced J. C. R. Lick Lighter as the head of DARPA's Information Processing Techniques Office, or TIPTOE. And also, by back then, DARPA wasn't DARPA, it was just ARPA. Uh uh, So this is the same group, by the way, that would end up doing a lot of work that would form the Arpanette a few years later, and the Arpanette

was the predecessor to the Internet in some ways. At least, the Arpanet was what ended up being the the building blocks for the infrastructure that would become the Internet. Now, all of that work happened after Sutherland had already departed the organization. His work became a fundamental component of both virtual and augmented reality. As I mentioned earlier, in nine he wrote a piece and essay. It's very short, it's very easy read, and you can find it online. The

title of the essay is the Ultimate Display. And if you ever do any research in virtual reality or augmented reality, this essay is going to pop up in your research, so go ahead and read it. It's like two pages long, so it goes very quickly. In that essay he talked about several ideas, including the idealized display, the ultimate display, something that would be the the furthest you could go

with display technology. Now, keep in mind, in his time, by his time he's still alive, by the way, but this time in the nineties sixties, he uh, you know, things were just restricted to monitors. You might have a light pen, but usually you just use a keyboard. Like it was pretty bare bones. They said, let's push this as far as we can imagine it. And in his example, he thought of a room that would be completely controlled by computers. Everything you would experience within that room would

be generated by a computer. Everything you see here, smell, taste, and touch, all of it generated by computers. The computer would even be able to form physical objects out of pure matter itself. Now, he wasn't suggesting that this would ever be a device that we would actually be able to build. He was just saying, what is the ultimate incarnation of display technology? And if you read it, you realize, oh, this is where the Star Trek Next Generation writers got

their idea for the Holidack. But unlike Star Trek the Next Generation, the Ultimate Display would not go on the Fritz every other episode and try to kill the crew. It was better than that. The Ultimate Display was sort of a a foundational like Philosophically, it was foundational for virtual reality and augmented reality. This idea of a very immersive experience where you, as a user, are surrounded somehow by this computer generated experience. And that's true both with

augmented reality and virtual reality. And augmented reality, the real world is still there, but you get this enhanced experience that is completely computer generated. So in nineteen sixty eight, Sutherland and a student named Danny Cohen would create a VR, a r head mounted display or h MD, and they nicknamed it the Sword of Damocles. Why because you had to suspend it from the ceiling. It was too heavy to wear on your head. You needed it to be

nice and sturdy. Uh. It included transparent lenses, which meant you could overlay computer information on the lenses themselves, and thus you could look through the lenses at the real world and have these wire framed graphics on top of

what you were looking at. And it also had a magnetic tracking system, meaning that it had sensors that could detect magnetic fields, and as you turned your head or you change the inclination of your head, it would change the magnetic field and this would be relayed as a command to the visual center the actual lenses themselves, so that it would the change would be reflected in what

you saw. So if you have a virtual environment and you turn your head to the left, you want the view within the virtual environment to go to the left too, but without head tracking technology that's impossible. So this was a very early example of head tracking technology. And again it used magnets magnetic fields in order to do that. Uh.

Obviously it's also really important for augmented reality. Again, if the a R system doesn't detect that you are looking around, then you're not getting relevant information, not for the specific thing you are looking at. Anyway, as I said, the graphics were pretty primitive. There were wire frame drawings, but they still showed that this was a viable approach to

technology using HMD for augmented or virtual reality use. Oh and one other note I should make so a lot of people say the sort of Damocles was the first head mounted display, and they say, you know, this is the first m D was made in nineteen I take issue with that. I don't think of the sort of Damocles as the first head mounted display. That to me should go to a different invention called the head site h G A D S I G h T now that was developed by Philco and unlike the sort of Damocles,

it didn't create a virtual world. Instead, the head site was sort of a remote viewfinder for a video camera. So imagine that you've got a camera mounted on a mechanical uh swiveling mount, so you can move it left right, you can change the orientation the inclination as well, and then you have that mapped to a head mounted display, so that if I put the display on and I looked to the left, the camera pans to the left. If I look to the right, it pans to the right.

That sort of thing it was meant to be away for for people to operate a camera in a remote location that might not be very friendly to a human being standing there. For example, the exterior of an aircraft. You could have a camera mounted on the outside of your aircraft that would allow an engineer on the inside to look around and maybe help a pilot land or navigate in a dangerous situation, or just get an idea

of the status of the aircraft itself. This was very much a technology that was being pushed by the military, an idea to create more military uses using this technology to make the military more competent, more adept at very rapid changing situations on the technology front, So headsite preceded the sort of damocles by about seven years. That came out around nineteen sixty one, but again it wasn't a virtual reality headset or an augmented reality headset. Was kind

of a like I said, a remote viewfinder. But still I consider that to be the earliest head mounted display, not the sort of damocles. However, Sutherland would end up going on to make lots of other contributions in computer graphics as well as the overall concepts that would guide both virtual reality and augmented reality development over the next

several decades. But now it'll be time for me to kind of move away from Sutherland to talk about some other developments that were important in a R. And before I get to that, let's take a quick break to thank our sponsor. All Right, we just left off with Ivan Sutherland. Now let's talk about a different father of augmented reality, Myron Krueger or Dr Myron Krueger. In nineteen four,

Dr Krueger create an augmented reality lab called Video Place. Uh. He was really into this idea of seeing the interaction of technology and people in artistic ways. He really wanted to explore artistic expressions using technology and people working together. So he wanted to create an artificial reality environment that didn't require the user to wear special equipment. You wouldn't have to put on a head mold display, or wear special gloves, or use any kind of device to control

your actions, because that's a barrier between you and the experience. Instead, his version consisted of a laboratory that had several rooms all network together, and each room had a video camera in it and a projector and a screen. Now, the video camera would pick up the motions of the person inside the room. It would send information to the projector, which would then project the person's silhouette on the screen.

And the silhouette was typically a really bright color, and you could move around and your silhouette would move around, So you almost became like a puppet master controlling your own silhouette. But then he started to incorporate other things, like other elements that were virtually on the screen. The projector was projecting things that were on the screen but not in the actual real room itself. So imagine a

ball and a ball is being projected on the screen. Well, you could move around so that your silhouette would interact with the ball and the ball would bounce away. That sort of thing, so you would be able to interact with virtual environments by moving around in a real physical space. And while those objects weren't really there in front of you, you could see the representation of them on the screen.

And this was really powerful stuff. And remember I said these rooms were all network together, so you could actually have a system where a person in one room and a person in another room both have their silhouettes projected together in their respective rooms on the screen, and your silhouette would be one color, the other person's silhouette would be a different color, and you could interact with one another.

And according to reports from this art experiment, they noticed that whenever people would have their silhouettes cross one another, they would actually recoil in their physical rooms. Keep in mind they're in different rooms, they're not in the same one together, they would recoil as if they had made physical contact or bumped into someone. So it showed that there was a very powerful psychological element to this virtual presence.

And again that psychological element plays a huge important role in VR and a R research and development, not just for creating products, but just to understand how we process information and incorporated into our sense of reality. Not to get too deep, for you guys. So experimentation in the field continued over the years. In the early nineteen eighties, Dr Krueger would write a book and publish it about artificial realities. But while the principles for augmented reality were established,

the technologies were still rather unwieldy. They were large, they weren't reliable, and it would require several years of work to improve those technologies, to create miniaturization strategies to get the elements down to a size that was more practical for that sort of use and wouldn't require you to have a head mound display mounted to the ceiling. And all of that took time, but you could tell that the ideas underlying augmented and virtual reality were already in place.

In n there was a Boeing researcher named Tom Coddell who coined the term augmented reality, and he was specifically using it to talk about this approach to overlaying digital information on top of our physical world to enhance it in some way. Now. Dr Cordell earned a PhD in physics and astronomy from the University of Arizona, and before contributing the term augmented reality to the public lexicon. He

did extensive work and artificial intelligence research and development. He also became a professor in the fields of electrical and computer engineering at the University of New Mexico. So when he was working with Boeing, he used this phrase to talk about specific system he was working on, an augmented reality system, and the whole purpose of this was to help people who were helping construct airplanes lay cables properly.

The whole idea was to use this system so that uh, an electrician can see exactly where the cable needed to go inside the partly constructed cabin of an aircraft, and that way you could follow the directions that you see through your display, lay the actual cable down where the guide tells you to go, and then you would have a properly wired airplane. Uh. And I'm sure, as we're all aware, properly wired airplanes are good airplanes. Improperly wired

airplanes are not so good. So it was a very important system to make this much more smooth and fast, and it meant that you didn't have to have as as many experts to guide the process. You could actually have someone come in who had never done this before and just follow the directions through this augmented reality set system and they could wire the airplane properly, so really

clever means of using augmented reality. Also, we would end up seeing that same sort of philosophy used again and again in the future in more sophisticated UH types of technology. It was the exact same approach, exact same idea underlying it. In Lewis Rosenberg proposed a system that the Air Force could use to allow someone to control devices from a remote location, and that consisted of a video camera which would provide the visual data to the user through a

head mounted display. They would wear the display on their heads or they would look at a screen, but typically they'd wear a display, and then they would also wear an exoskeleton on their upper body that would allow them to control some sort of robotic device, typically robotic arms.

And usually the way this would work is that the display was designed in such a way with a video camera so that the view that the person had it made it look like the robot arms were their actual arms, which required a little bit of trickery on the part of Rosenberg. They had to fudge the distances between the video camera and the robotic arms to give this this sort of feeling that the robot arms represented your actual arms.

So you move your arms inside the exoskeleton, and the robot arms would move as well at the remote location, so you it's kind of like a really fancy remote control. Now imagine that the robot arms are holding various tools. Uh. The suit would also provide haptic feedback, that touch based feedback to let a user know more about what is

going on when they're operating the arms. So if you were to have do something that would make a robot arm encounter resistance, then you would feel haptic feedback in the suit that would indicate, oh, you're you're going beyond the parameters of where this robot arm is capable of going. So you learned very quickly where where you can operate within that suit and make sure that you are not pushing it beyond its limits. You could also uh end up using these tools to do various things in this

remote environment. Now, Rosenberg called his system virtual fixtures, which meant that the user would see these virtual overlays on top of a real environment that they were looking at. So I'm going to give a very basic example that to illustrate this. Because it's hard to imagine, it's hard to get it across in words. But let's say you're looking through a headmund display and in front of you is a board, wooden board, and it's just a regular

wooden board. There's nothing painted on it or anything in the real world, and it's in a room that's across the building from you. You cannot see this with your own eyes. You can only see it through the video camera. The virtual fixture overlay might be a series of circles, and the circles are things that you were meant to cut out of the board using the robot arms and a tool that's right there inside the physical environment across

the building from you. So you follow the patterns that you in this virtual overlay and you complete the task. That's a very simple example. And uh, this system was meant to allow for that. That's what he would call the virtual fixtures, these overlays that you would see that would appear to be real, but actually we're not present

in the physical environment itself. Now. Also in a group of researchers at Columbia University were proposing a system that they called the knowledge based Augmented Reality for maintenance assistance a k a. Karma cute. Their approach was pretty novel. They pointed out that while augmented reality had tremendous potential, it also had a really big barrier, and that it takes an enormous amount of time to design or animate

and implement these graphic overlays for a our applications. So let's say you're in a room and you're looking at different objects, and little labels are popping up for each object. If you're having to do all that by hand, it takes a huge amount of time. What they wanted to do was create artificial intelligence systems, or at least techniques

to generate graphics automatically on the fly. So this would be similar to using image recognition software, so that if you look at a specific box, let's say, the image recognition software might be able to map that box to a specific product and thus give you an overlay of information about the product that would be inside that box. And it would be able to do all this automatically. It would not require a human programmer to go through and and look at every single product in every single

type of box and program all that out. That would be ridiculous. It would take forever so it was the work of this group with Karma that really started the ball rolling with this AI approach to automatically fill in that information and make a are a more practical experience. Around this same time, between ninety two and nine, the Laurel Western Development Labs, which was defense contractor, began to work with the U. S Military to create a R

systems for military vehicles. And you can understand very quickly how a R would have enormous potential for military applications. And in fact, a R is very commonly used in lots of different things, like pilot helmets, where it helps pilots keep track of targets uh and identify potential threats, that kind of thing. But in this case, they were really looking at creating a augmented reality system that would create virtual opponents for people working in simulated wartime conditions,

so really a training program. Imagine that you're operating an actual military vehicle like a tank, and you have a view outside that is really an augmented reality system, so you're actually looking at the real world around you. You

aren't just sitting in a simulator inside of building. You are out there in the field controlling a real vehicle moving around in real terrain, but you also see virtual representations of enemies in that real terrain, and you can practice maneuvers and firing on enemies that sort of thing, probably not using live ammunition at that point, but having a more realistic simulation in a real environment, so that

you're not just trying to create a totally virtual scenario. Anyway, that work was done in the world wouldn't really learn about it at large until about ninety nine, because that's the way the military works. They're not so eager to talk about their stuff while are still doing it. Meanwhile, at the same time, artists were continuing to explore the relationships between physical performers and virtual elements. You remember I

talked about Dr Krueger earlier. While in different artist Julie Martin would create a piece called Dancing in Cyberspace, and in that piece, dancers on a physical space or a physical stage. We're able to manipulate virtual objects, so an audience would be able to see both the physical performance by the dancers and the virtual reactions the things that happened within the virtual environment as a result of the

dancers moving around their physical space. Pretty neat two researchers Reki Moto and Nagao created there the first real handheld a R display, but it was a tethered display. It wasn't free form. You couldn't just take it anywhere. It was called Navy Cam, and you had to have a tether like cable essentially connect the Navy Cam to a workstation.

But it had a forward facing camera and you can use a video feed to go through this handheld device through the cable to the workstation, and it could detect color coded markers in the camera image and display information on a video see through view. So you can get that augmented reality experience. Obviously very limited, you know, you could not just carry this around with you everywhere you go, but it showed the ideas behind augmented reality could in

fact be realized in a handheld format. Now. It was just a matter of getting those different components small enough to all fit in a self contained mobile form factor. In the late nineties, we started seeing televised sporting events featuring augmented reality elements, or at least you did. I don't watch sports ball, that's not entirely true, but I don't watch football or hockey. American football or hockey, and both of those were the sports that really got them.

First off, I'm gonna backtrack. I used to watch hockey, but then Winnipeg stole the Atlanta Thrashers from me. Winnipeg, Okay, getting back to hockey. So hockey had the Fox track system, which Fox put into hockey games so that you could easily follow the puck. Instead of trying to watch this little bitty black disc spinning around, you got to watch this very bright, highlighted neon colored disc that everyone hated. And after about two seasons, Fox stop doing it, and

people were happy. Untila Thrashers moved away, and then it was just miserable. American football would follow suit in the late nineties and have the first down line introduced, where they could on live video overlay the first down line. Usually it's a bright yellow line that indicates how far the offensive team needs to go. And by offensive, I mean they're on the offensive. I don't mean they offend

my sensibilities. I'm not that against American football, but it showed how far they would need to go in order to establish a first down, which I am told is something you want to do. Uh. That would start to get employed in and over time we would see that increase where eventually Skycam was able to even use this system. At first, it wasn't you could get a Skycamp you but you couldn't do the overlay of the first intent

line until later. Well, I've got a lot more to say about augmented reality, but before I do, let's take another quick break to thank our sponsor. Okay, we're back. Let's skip ahead to I guess it's not really skipping

I just talked about. Let's plot ahead to That's when NASA's X spacecraft was using an a R system as part of its navigational tools, so people back on Earth could look at a view from the spacecraft a camera mounted on the space ace craft, and on top of that view they could overlay map data to help with navigation. And all of that, of course was controlled back here on Earth, but it was sort of an experiment to see how augmented reality could be incorporated into space exploration

missions in the future and make them more effective. Also, in the Navy began work on the Battlefield Augmented Reality System or BARS, which is a wearable a R system for soldiers. You've probably seen various implementations of this over the years is obviously evolved since nine It's one of those pieces of technology that some soldiers took to but

a lot just felt that it created unnecessary distractions. Technology and warfare is very, very difficult because there's sometimes where we think, oh, more information is always better, but in some cases that doesn't seem to hold true. Uh, And for some people with these head mounted displays or or really it's heads up displays HUDs uh, that can sometimes be the case, depends on the implementation. In two thousand, hiro Katsu Kato created a software library called a R Toolkit.

Very important software library was also open source, so anyone could contribute to it, modify it. Brent put out a new version that sort of stuff, and it uses video tracking to overlay computer graphics on a video camera feed, and it's still a component for a lot of a our experiences today. Later on in the two thousand's this would be adapted so that it could also be used in web experiences, not just native experiences to specific devices, and we continue to see are built into new experiences

including smartphones and tablets. By two thousand four, some researchers in Germany, we're creating a r apps that can take advantage of a smartphones camera. But two thousand fours pretty early for smartphones. It really would would be a few years before this would truly take off, because that's when Apple came out with the iPhone in two thousand seven.

That was the real revolution in smartphone technology. There had been smartphones before the iPhone, don't get me wrong, and many of them were really good, but the iPhone was something that caught the public's attention and made smartphone sexy. And because of that, there was a ton of money poured into the smartphone industry as well as not just Apple, but also to other companies, like the companies that were

offering Android smartphones. But I think we can really thank Apple for all of that happening in the first place, especially things like seeing that accelerometer where you could switch from portrait to landscape mode. I remember everyone freaking out about that when Steve Jobs showed it off in two thousand seven at Macworld, and everyone thought, will this is the this is amazing. Well, we take it for granted now,

but it was a big deal then. So once that smartphone revolution happened, it was a landslide victory for both augmented reality and virtual reality research and development, because it meant that so much money was being poured into creating newer, thinner, more capable smartphones that we saw an explosion in technological development that could also be used for virtual and augmented reality experiences. So, for example, I think of those sensors

I talked about earlier, accelerometers and gyroscopes, that sort of thing. Well, we saw a lot of development in those spaces in order to make smartphones better, and people who are working in a R and VR experiences can take advantage of those same sensors either creating apps specifically for smartphones. Thus, you don't have to build any other hardware, you just use existing hardware. But that limits how you can use it, right because you don't typically wear your smartphone directly in

front of your face. Or they could end up taking advantage of those new, smaller sensors and incorporate them directly into brand new hardware like various types of wearables like Google Glass for example, but that would be a few more years. In two thousand and eleven, Nintendo launched the

Nintendo three D S, which included a camera. It was you know, the three D capable handheld device and included actually a pair of forward facing cameras so you could take three D photos if you wanted to, and it also had some a R software included with it. You would get these special Nintendo cards kind of like playing cards, and if you were to point the camera of the three D S at the card and look at the screen, you would see a little virtual three dimensional character pop

up on the card. So Mario would be an obvious example. You put the Mario card down on the table, you hold up the three D S, and you aim the camera at the card, and you look at the screen and there's Mario, and Mario appears to be jumping around on your physical table. Now, obviously, if you look off of the display, there's no Mario jumping around, but on the display there he is, and it was pretty cute.

I remember being really impressed with this very simple implementation of a R when we got our three DS, and then I took our three DS apart, and then I took pictures of it, and then I posted on Twitter and people got sad, it's a great day. In two thousand thirteen, Google introduced Google Glass that was the wearable that included a small display position just above the right eye. Uh, So when you look straightforward, you could tell that there

was something kind of above your natural eyeline. But it get in the way too much. You too to look at the screen. You actually had a glimpse, You had a glance upward, and then you could see what was on the display. Google Glass had augmented reality features like crazy. You could see video calls. You could actually use the glasses to not just to take a video call, but show the other person what you are looking at so

they could see from your point of view. You could also overlay directions, so if you're walking down street, you could glance up at the screen and it would tell you if you need to keep going straight or turn laughter, turn right, that kind of thing. It was really useful. Um. I had a pair of these Google Glass and I

really liked the direction they were going in. I felt that it wasn't a fully realized product at the time, and eventually Google agreed and after a couple of years they took Google Glass off the market entirely, and now you can't get them anymore. Uh. They were clever, but they were expensive and they had some limitations, and like I was saying earlier, you know, it's hard to build

all the components you need into one headset. So Google Glass would communicate via Bluetooth to your smartphone, and your smartphone would act as the actual nexus point to the Internet. But it was a neat idea, uh, and I enjoyed getting to use them while I did, so I keep hoping to see a return of that kind of technology, but perhaps in a more mature and less expensive format. Now we've also seen applications similar to the ones we mentioned earlier, the ones that are meant to guide people

into laying out or repairing a system. We've seen that in the car world. Not too long ago, there was the MARTA system introduced by Volkswagen. MARTA makes me chuckle because that's also the name of Atlanta's public transportation system, But in this case, it stands for Mobile Augmented Reality Technical Assistance, and it's specifically designed for mechanics who are

working on the XL one vehicle. So if you hold up an iPad that has this app on it, and the camera has pointed at an XL one and you look at the display, you'll see information overlaid on top of the car, including labels for all the different parts. So let's say you're a mechanic and you have to do a specific repair on this vehicle. You hold up the iPad, you look through the display, and you see exactly what you need to do. It gives you a set of instructions that shows you how you need to do.

It tells you where you need to stand based upon the angle of the view. So if you hold it up and it says no, you need to move about a foot to the right, you can do that. Then hold up the iPad again. I'll say, all right, you're in the right spot. Make sure you loosen this particular bolt first, that kind of thing. And it's meant to be an interactive maintenance guide in a way maintenance and repair guide. This is one of those applications of augmentary

lity I think is a no brainer to me. It's a killer hap the idea of having an ability to work with something you are not familiar with, but you're able to leverage the expertise of people who either designed it or built it, or just fully understand it and get guidance based on their expertise in real time, so you're not having to go and consult a an article about it, or watch a YouTube video. You get step by step instructions overlaid on top of your view of

that thing. To me, that's the most compelling use of augmented reality from a practical standpoint. There are a lot of other uses that they'll talk about towards the end that I think are also really super cool. So don't get me wrong, it's not the only one. But let's move on to two thousand fifteen. That was when Microsoft would unveil the hollow lens, something I still want to try out. I have not had a chance to try

a hollow lens yet. That is a headset capable of advanced a R applications everything from what I was just talking about, giving you guidance, step by step instructions on how to do like a repair job on say an electrical outlet. You can even use a Skype system to call an expert who can then view your point of view and interact with that point of view. So let's

say I'm looking at the outlet. The expert electrician I'm talking to can see what I see, and he or she can also make notes on the display which shows up in my field of view. So he or she might circle a specific wire and say you need to you need to remove that one first, and I know I need to do that one first because I can see which one they are talking about. Or they might circle another wire and say, no, matter what you do,

don't cut this wire. Well, the toilet ups stairs will explode like lethal weapon too, and I won't do that because you know that guys like three days from retirement. So I have a heart. But no, this is this is a really neat idea, having this interactive ability to overlay the information from the world, the digital world, onto your physical world. And beyond that, the hollowns has lots of other functions. It's not just something to do, you know,

home repairs around the house. You can also use it for entertainment purposes, like you could create a screen that can show you video from various sources and you can assign it a place on a wall in your environment. Let's say that you're in your living room and you just create a screen so you can watch Netflix, and you slap it on a wall and it will stay in that same position relative to your point of view.

So if you look to the left or right, the screens days where you put it as if it were physically they're on your wall, but keep in mind it's just a virtual screen, and when you look back to that part of your wall, you'll see the virtual screen there playing whatever it was that you wanted to watch.

I think that's a super cool idea. And they've also shown off games like a game of Minecraft that uses hollow lens so you can actually view a Minecraft world sitting appearing to sit at any rate on top of a table, so you can walk around the table and view the Minecraft world from various angles and play that way. I think that's super neat. Don't know how compelling it is, because again I haven't tried it myself, but I really

like the idea. This year, two thousand and sixteen, are got another big boost from a little game called Pokemon Go. Although I have to admit this was a really primitive, basic implementation of augmented reality. Really, it was not much more than just a In fact, it was nothing more than just an animated overlay that would exist on top

of the camera view of your of your device. So I'd say I'm holding up my smartphone and I'm trying to catch a Jiggli Puff, and the Jiggli Puff is currently bouncing up and down on the sidewalk in front of me. That's about as far as the augmented reality

actual experience would go. So very primitive. But because Pokemon Go became so popular so quickly, it really pushed the concept of a R to the front of the minds of people everywhere, including business owners who immediately said, we need an augmented reality app Whether they actually needed one or not is beside the point. A lot of people got into a are because of Pokemon Go, uh, for both good and bad. I always think that you have

to come up with the experience first. You have to understand why you need to use a specific strategy to create a specific experience, and then build it. Not hey, we need augmented reality makes something that's a R. To me, that's the backwards way of going about it. But what do I know. I'm not not a programmer, so I'm sure the programmers feel in a similar way to me. But that's just a guess. Now. The future of a R depends heavily upon the applications we see in which

ones end up being successful and which ones aren't. Right now, I would say that the best bet is to see more a R features built into smartphones and tablets. Uh not, maybe not necessarily built into them, but have apps available that create a R experiences for very specific contexts. Like

let's say it's a museum app. You might download a museum app on your phone, and when you go to the museum and you use your phone, you can get more information about the paintings and sculptures and other installations that you see in the museum. That's an easy one to understand. But that same app isn't going to be useful once you leave the museum, you no longer have

the context that it is tied to. I think that smartphones are probably going to be where the greatest development is going to be in the near term, because wearables is still really hard to do. We still don't have a consumer version of the hollow lens out available for anyone to purchase, and it may never come out as a consumer product. Microsoft hasn't shown a whole lot of

interest in making it a consumer product. Maybe that will change, but at the moment, I wouldn't hold my breath, so I would argue smartphones and tablets are pretty much where it's at maybe some implementation with some existing VR headsets which have external cameras mounted on them as well, like forward facing cameras. You could build a our experiences there.

Then it gets a little weird because you're you're also you know, you're looking at a monitor, so you're looking at a video feed of your surroundings, and on top of the video feed, you get the overlay. Same thing is true for your smartphones and tablets, by the way, but different that from the Google glass implementation, where you're looking at the actual physical world, not a video representation

of it, but the real world. And then because the display itself that you are looking through is transparent, you're looking at a transparent overlay of digital information that gives you more info about the world you are in. I think a R is super cool. I think it's really got a lot of potential to change the world around us and to change the way we interact with the

world around us. You could imagine a dystopian future implementation of a R where we all have to wear glasses and we're constantly getting personalized commercials beamed at us whenever we look at anything, like imagine walking past a store casually looking in the window and then getting a whole bunch of ads for all the stuff that's in the

store window. That would be obnoxious, and it's easy to understand how people would not want that, yet also easy to understand how that could possibly become a few her or think of the future where your privacy is no longer even relevant, and you walk down the street and you look at all the people's faces who are also walking down the street, and you're getting names of everybody

and what they like and what they dislike. You know, what music they tend to listen to, maybe what they're listening to right now, And it's all because we've got facial recognition technology. Almost everyone has some sort of social media presence, so you can map that face to any public profiles, try and find a match. If you found a match, you can bring back information to the person

wearing the glasses. So I can look at somebody and say, oh, this, uh, this cute kid over here, she's got she likes punk rock music. I'm gonna I'm gonna impress her with my knowledge of the cramps. That probably wouldn't work, but the point being that's pretty creepy and invasive, and so there are some negative implementations of a are that we have to watch out for unless we get to a point where we just don't care about privacy at all anymore.

Some would argue we're already there, and in that case, this implementation of a R may not sound creepy at all. It might just sound kind of cool, kind of the equivalent of walking into a store seeing a person with a name tag and addressing them by name. If they don't remember they have a name tag on, they have this moment where they think, do I know you? That.

If we're in a world where everyone can see everyone's name all the time, then well, for one thing, I won't ever have to worry about coming at a loss when I have to introduce my wife to someone. So that's a that's a plus side. Okay, I'm in favor of it now, all right. Well that kind of wraps up this deep dive on augmented reality. There's so many

different other things I could have mentioned. There's so much more to the history and development of it and all the pieces of technology, But this is one of those strategies that arose because so many other elements had matured over a great deal of time. If you guys have any suggestions for future episodes tech Stuff, whether it's a specific kind of technology, or how technology has affected us,

or how we affect tech. Maybe it's a specific company or person in technology, or you just have a suggestion for somebody I should have on the show, either as a guest co host or someone I should interview. Let me know, send me a message. The email to send it to is tech Stuff at how stuff works dot com, or drop me a line on social media. You can find me on Twitter or on Facebook with the handle tech stuff h s W and I'll talk to you

guys again really soon. For more on this and thousands of other topics, is it how stuff works dot com.