How Eye Tracking Tech Works

to cover on this show. And in the past I've covered concepts like Moore's law, and that can tend to lead people into thinking that, because computer processing power effectively doubles every two years or so, that the entire world is going to be transformed into a crazy techno wizard world that you'd see in an imaginative science fiction film. It's gonna happen any day now, and we're all headed

that way. It's a foregone conclusion. However, from a day to day perspective, it doesn't seem like we're going all that quickly, despite that amazing progress and computer process using power. And one of the reasons there are many, but one of the reasons for this is that while computer processing power tends to follow that pathway that Gordon Moore observed several decades ago, other aspects of technology aren't on quite as predictable a path, nor do they advance at such

a reliable accelerated rate. And because of that, because some aspects of technology lag behind the development of processing power, we don't get that crazy future that we've been promised since say the nineteen fifties. Now, one of those aspects, one of the things that does not evolve as quickly as processing power, would be user interfaces. Now, these are the ways that we interact with our devices, such as our computers. So the typical computer user, they're usually relying

upon your basic keyboard and mouse combination. It's the same thing that's been in place, say the nineteen eighties and personal computers. Xerox Park was introducing things like the mouse and the graphic user interface much earlier than that, but it was really the Apple Macintosh that brought the graphic user interface and the mouse along with the keyboard to the personal computing space. The keyboard, of course, had already existed. Uh,

there's a problem with this type of interface. There's actually a couple. But one is that it requires users to be both able to manipulate the input devices, which would be the keyboard and mouse. You have to actually be able to move them in order to interact with your computer. And the other problem is that there's a learning curve. You actually have to learn how to use those devices in order to interact with your computer properly. It's not a natural way to interact with a machine, or with

anything else. It's not the way we tend to interact with the stuff in our everyday world. We don't have to type out codes or move a random part of an object in order to make something else move. We are much more direct with our input whenever we are interacting with our environment. So in order to learn how to use a computer, we actually have to go through this process of learning how keyboards and computer mices work.

Otherwise nothing much gets done. Now that's not to say that once the keyboard and mouse combo came out we stopped trying to find ways to interact with technology, because engineers have been working on everything from light pens to touch screens, to gesture controls to voice commands all the way back to when the keyboard and mouse started to become a thing. And this is all in an effort

to broaden the way we control computers and electronics. And I've covered some of those technologies in earlier episodes of Tech Stuff, and I've explained that many of them, like voice recognition do not follow a path that's nearly as accelerated as processing power. We've gotten really good with voice recognition over the last say, five or six years, but for many years it was lagging behind other technologies. And this is the problem with making those big promises about

what the future is going to be like. Often that is based on an assumption that everything is moving forward at the same speed, and that's just not true. Well, today I wanted to cover a specific type of technology that can be used in user interfaces, although that's only one application of this technology, and that is eye tracking technology.

A bit later in this episode, I'm going to play an interview I had with Oscar Werner of Toby Tech, and Toby Tech is a business unit of a company, Toby that is pioneering eye tracking technology for all sorts of applications, both on the business to business side and

the business to consumer side. But before I get to that interview, I thought it might be interesting to talk about the evolution of eye tracking in general, because it's actually a really cool story, and you I learned stuff I just didn't know, including stuff that schemed me out a little bit. So I want to share my skiviness with you. Wait, no, that's not right. I want to share the stuff that schemed me out with you. My

skiviness is a matter for my own personal life. Back off alright, To begin with, eye tracking technology is the product of a few different disciplines, and they originate from different perspectives. You've got philosophical approaches, you have technical approaches, you have biological approaches, and you also have human behavior along with psychology and physiology. So in the eighteenth century, which I feel I need to point out, was an era that had very few personal computers in it because

they had not yet been invented. There was a guy named William Porterfield, who should not be confused with professional cricket player from Ireland, totally different person. This was William Porterfield in the eighteenth century, an Englishman who made some general notes about oculomotor behavior and reading, in other words, how our eyes behave as we read text. Now, Porterfield's work was not based off empirical evidence, but rather casual

observations and some hypotheses. But later scientists would study this behavior in greater detail and with more rigorous scientific approaches. So some of them looked at it from a physiological standpoint, some looked at it from a philosophical standpoint. Some of them combined the two, and a man named Louis Emil Javal conducted a study of how our eyes behave when we read text. He was working in the late nineteenth century,

around the eighteen sixties and eighteen seventies. He published his work about the US in eighteen seventy eight, and he made his observations by directly watching people's eyes as they read lines of text. And he saw that when you do that, your eyes don't just glide seamlessly across a page. Instead, your eyes have lots of stops and starts. Those motions he gave names, so he called every time you stopped, when you would your eyes would pause as they went across,

he called those fixations. And he would call all the bits where your eyes would move across rapidly Saccadays, those would be the starts. Now, saccad comes from a French word that was originally used to describe horse movements, and a rough translation into English might be jerk, as in the motion, not as in some jerk face who's giving you a hard time. So Javal's work was interesting in that,

for the most part. The only studies in oculu motor function up to that time were limited to diagnos sing and perhaps treating dysfunction, so the work wasn't all about learning more of what typical behavior is like. In other words, doctors normally didn't think to look into the way our eyes move unless something atypical was going on with the patient. Javal started a movement in medicine and biology that, pardon

the pun, shifted the focus. Other scientists observed similar similar oculomotor behavior, so this was all emerging around the same time. There are actually quite a few people who were looking into it, so to speak. Javal just became the most

famous of all those people. So I don't mean to suggest he was the only one, but rather he's the one that gets most of the credit, and that is because of another person named Edmund Huey, who was a clever fellow from the late nineteenth and early twenty centuries. He made more progress in this study by creating an invention that sounds like it came straight out of a

clockwork orange. It consisted of a plaster I cup that would fit over a subject's I so it would go right up against the person's eye, and the cup had a lever attached to it that went off to the side. At the other end of the lever there was a pen, and the pen could move across a rotating smoked drum, and thus the pen would make marks across the smoked

surface of this drum that rotated around. And the idea was that this would allow you to look at those marks and map that to I've movements that are happening while you're actually giving the subject a test. So as the subject's eyes would scan text, the movements of those eyes would make the plaster cup move, which would transfer that motion to the lever and ultimately to the pen

that was resting against the rotating drum. And this sounds absolutely ghastly to me, but it gave valuable information to Hueie about how people's eyes move when they're reading. Now, in his papers on the subject, hue would refer back to Javal's work and his research on the physiology of reading. There were other scientists who are looking into this subject as well, but Huey's work and the citation of Javel have led many to simplify history and just say they're

the ones responsible for the research. On i've movements in the early days. Later on, another scientist, George Malcolm Stratton, would start to study how we move our eyes as we perceive illusions. Now, this is really interesting. Optical illusions depend upon our eyes perceiving something that is either not there or is there in a way that is different from the actual physical thing that we are perceiving, and therefore the way our eyes behave is very important to

how those illusions work. What is it that's going on physiologically with our eyes, how much of that is dependent upon my behavior versus what's going on in our brains. This would slowly lead scientists to the conclusion that our eyes are really an extension of our brains, and that the process of thinking is very closely, if not synonymously, connected with the process of seeing. And I think that's

pretty cool too. Stratton's work would publish in the early twentieth century and also help inform scientists and others about the nature of our eyes. All of this is to say, these early studies began to coalesce around that concept of not just how our eyes move as we perceive, but also where our focus tends to be directed. Knowing about focus is important for lots of different fields, some of which appear to be completely independent of technology and user interfaces.

For example, let's say that you are a graphics designer and you have a job. You're given a contract, and your job is to create a sign that will warn people to potential danger. Maybe it's a warning sign for let's say radioactive material, and you want to make sure you get across to people that anything beyond that sign could pose as a threat to that person's health. There are many things you would have to consider before you start making that sign. For example, which colors should you use.

You're gonna want something that's going to have a good level of contrast, so that whatever the message is that you're trying to convey will be easy for people to perceive, even if people might have some form of color blindness. Then you have to ask, all right, well, if there are any words on there, what typeface should you use? I mean, it needs to be easily legible, and where do you draw the focus on this sign so that the message you are sending gets to your audience and

they actually get the point. Being able to track eye movement gives us more information about how we humans work, not just how we can interact with stuff. As for eye tracking technology, it thankfully evolved beyond the need for eye cups and mechanical levers. Otherwise I would just be running around screaming inside of the studio because I have

a thing about eyes. But one methodology still relied on the subject wearing special equipment, and it was a system consisting of a pair of contact lenses with a sort of special feature, which is dependent upon the actual implementation

of the technology. Some versions would use contact lenses that had a magnetic field sensor connected to them, or they might have an embedded mirror that would reflect light, and then you would have an external system that has partnered with these contact lenses and that would help monitor the movement of the special component of those contact lenses in

turn that would track eye motion. This typically would provide a really accurate indication of what the eye is doing at any given time, assuming that the contact lens fits properly and doesn't itself slide a bit across the surface of the eye. Obviously, if that happens, then it's messing up your metrics because you're looking at the movement of the contact lens, not necessarily the movement of the eye itself.

But for something like a computer or a mobile device that a consumer might purchase and you want an interface for that, you probably don't want to deal with putting in special contacts. So the interfaces that we're finding with personal electronics and computers typically rely upon optical tracking. And that means exactly what you think. It means. You're using some form of camera to monitor where you are looking.

The device uses cameras that can detect your eyes using software that has image recognition algorithms running in it, So it's looking for different aspects of eyes that are common across many people. And once it identifies, all right, that's an eye, then it starts to try and track where the eye is looking. Typically, you have to end up using a system to set a baseline for this. You can't just turn it on and start looking. You have to calibrate it. So you calibrate the system so it

knows where you're looking. It might guide you into looking at specific regions of a screen, whether it's a mobile device or a computer or whatever it might be. And that way, once it knows quote unquote that you are looking where you're supposed to, it can then uh interpret where your eye is looking from that point forward, because

it's calibrated from a known set of standards. And from that point forward, assuming that everything is working properly, every time you look at some other part of the screen, the eye tracking software can track your gaze and know exactly where you are looking. That's the basic idea behind it. Now, when we listen to in on an interview in just a few minutes, Mr Werner will explain how Toby's technology accomplishes this. In particular, they have a specific kind of

approach to using cameras to track eye movements. It's not that different from the way motion sensors like the Microsoft Connect work, only obviously, eye tracking technology is more precisely tuned to look at the movement of your eyeballs, not your overall body. Now, this technology allows for all sorts of potential applications, and some of them are passive, which means that people can use the eye tracking data to get an idea of where people direct their focus for

any given situation. So that might be when someone is looking at an advertisement, or looking at a web page, or playing a game. It can be more active. It doesn't have to be a passive implementation, it can allow people the chance to send input into a computer by staring at specific parts of a screen. This is useful for those people who may otherwise be unable to move. They can use these kind of interfaces in order to communicate. And there are countless other applications that exist now and

even more that we haven't even thought of yet. To get a better idea of what this tech is all about and how it might be used in the future, I talked with Oscar Warner. He's the business unit president of Toby Tech, and we'll hear that interview right after we take this quick break to thank our sponsor. Let's serve by you introducing yourself and your title and the company.

So my name is Oscar Warner. I'm president of Toby Tech, and Toby Tech is basically one of three business units in Toby U in Toby Group, so um Toby there are three business units. Toby Dinox is providing solutions for

people with disabilities. Someone with a S who cannot move, any who cannot move at all can basically communicate via looking at a computer or send emails via just looking at the computer, or somebody with CP who is pastic can type emails and write on a thesis or or or or write on a computer and use a computer just with the rice. And then it's Toby Toby Pro which is focusing on selling research people's solutions to to universities and to companies like Proper and gaming the U

in the level to understand human behavior. And then it's Toby Tech who is the developer of all the core eye tracking technology and is selling this to the consumer markets, which may be PIEC, gaming, the PC industry, smartphone industry, v R, a R industry, so selling to the big big o emsum of which you know pretty much all the names. Right, So let's start just by kind of explaining what I track tracking technology does in general, so without getting into the mechanics of it, what exactly is

its purpose as far as as a technology goes. So, eye tracking does two things, um um. It basically makes technology understand you as a human being, and it does that in two different ways. You know. The first one is what we in already call insight. It makes devices know what you're paying attention to, and when you think about it, what you're paying attention to is your interest

or your intent at that given point in time. So a computer with eye tracking know what your intention is and when you think about it is if you think on your in front of computer, you're looking at an icon that you want to click on. The first thing you do is you look at it. The second thing you do is you take your mouse and drag the mouse and throw over it. And the third thing you do is you click on your mouth or in your

touchpad or in your control or whatever. But a computer with an eye tracker already know that you're looking at something, so that it knows what you want to click on before you touch your mouth. So that could come in really handy for anyone who's designing something to have a process where you run the test and you have eye tracking technology, you can see what elements of your design are actually working or maybe are distracting to your to

your potential user. Yes, that is exactly what to be pro that their entire business models is centered around that. For example, you know they're creating glasses eye tracking, a set of glasses and they send a test group through a store, a retail store, and then big retail companies such as Propering annulal unit level that would test the store layout and see what package did I look at? What package did I not see? And how do I reconfigure the store layout. It can also be done on

websites or pack research, etcetera. But US excellent and it can also be used not just to see where our attention is, but it can go a step further and become a direct interface with some other technology. For example, you can use it so that you are creating an action, you're removing that need to move and click correct. Yes. So the first thing, the first category is just it gives computers or it gives devices insight of what you're

looking at. If you're in front of the computer, are you're focusing on the screen, or you're leaving the screen, what you saw, what you did not see. It can be presence and identity in all these things. It gives you that it gives you another knowledge about the human. And the second thing is you can use it for

interaction like taking an example from VR. For example, if you imagine you pick up something in something in your hand and you want to throw it, and if you do this, you throw it aim at the a's in your room. You want to throw it. When you as a human aim at that place, you are looking at the place with your eyes and then you give the command to your body to throw the thing with your hand.

But the VR headset in order to tell where you want to throw the object, you either need to point at the area you want to throw it it with your forehead or you need to point at the area you want to throw at it with your hand or do a gesture, and that's very very hard to get exact basically, So this is kind of replicating the human behavior when you pick up pick up something and you want to throw at something, you look at the point you want to throw at and that's how you do

your aim. And headsets with eye tracking would suddenly understand your intention. They would understand what you are trying to

hit because you're looking at that point. So those devices are much smarter, and we give developers the freedom to use that and it removes that barrier from a user perspect active that any time you have any kind of interface with a machine, the more levels of of abstraction that there are between you and what you're trying to do, the more there's a learning curve, right that you have to actually train yourself how to use that device, so that the device does what you wanted to do, whereas

this is kind of stripping that away quite a bit. We're trying to make it. We're trying to make it natural. That is exactly how you behave in the real world. You want to throw an object at something, you pick it up, you look, that's how you aim and then you throw, while in VR you would have to do something else. So so we're trying to make it as natural. Yeah, So what we think when you think about what devices are as of today, devices have and I'm talking about

devices a community or a or smartphones or computers. But they they they know if you push the buttons or push them right. They listened to you because they have cortona or they have different types of voice input and microphones. They know where you are because they may have a GPS. They know if you turn them around because they have a gyro, etcetera. They know all these things, but they have no concept, no idea of of the human in front of them. They don't know if there's there's anyone

in front of them. So we typically say that devices as of today are blind. They can listen, they can feel, they know where they are, they know in what direction they are, but they are blind. They have no concept of who is in front of them, and what is that person doing, and and where is that person looking what is that person's intention? We think that's in the in the world of AI, you know, trying to create machines that understands use humans. We think that's kind of

actually crazy. Um, it is I would I would probably argue pretty strongly for that. It is it really or more important to know what is your what is my user's intention? What are they interested in at every given point in time, and knowing if they are in front of the computer, as knowing where they are or in

what direction the laptop or smartphone is turned interesting. So with this, this philosophy, this and this general approach, this idea of teaching machines how we humans pay attention, you know, looking for the signs that show that we are in fact focusing upon any given aspect. What is the actual technological component that allows these machines to do that. I imagine that we're talking about a hardware side with a camera and then obviously a software side that's processing that

information and making meaning out of it. Yeah, I mean the core components of a of an eye tracking system. It's it's a it's a camera in near infrared camera and it's a set of illuminators that can that protected light pattern or your eyes, and then this camera takes

the pictures with this light pattern. Then you send these pictures the picture stream onto sometimes formal processing units where there is a set of algorithms that can calculate um either where you're looking, or you can do face idea on the same camera, so who you are, or it can calculate what direction your head is. It can detect your facial pictures. Are you happy, are you sad? It can take the drowsiness, are you awake or sleeping, So you can do a lot of different things with those pictures.

And then the third component, it's kind of the camera, the algorithms, and then it's the use case. So what do you do with the knowledge of that you're looking here? That's kind of built into the use case, and what do you do with the knowledge that you know the identity of this person? So it might be something in the case of authentication where you're looking at a person's idea,

their their facial structure. Similar to what we saw with the the reveal of the new iPhone handsets that are coming out, they're using a face i D technology for that sort of thing authentication, but it could it could also be control systems, whether it's in a game or as you had mentioned earlier, for people who have mobility disabilities that they aren't able to move, they're able to perhaps use their eyes to focus on specific icons and

thus communicate using eye tracking software to indicate what they're intent is. Yeah, so you see big trend in consumer actronics right now. If I mean the iPhone is is just one of them. They're entering because they want to make their devices smarter and understand the users better. They enter special use of facing cameras, nearing for red cameras

which can understand their users more. And that is exactly what iPhone did, I mean iPhones quote in one of the newspapers or articles I read was what can be more natural than touch? Well just a look. So iPhone Is or Apple is recognizing that, hey, yeah, we're using the touch interface today. But if I really won't understand my user, I need to understand what the user is, what the user is doing in front of advice. That's why they implemented face i D sure in the phone um,

And that's why I mean Huawei. We did the phone with Huawei Run about a year ago which had a similar type of similar type of camera which could do various features as well and understand you as a user. Are you in front of it? Then the screen didn't live, etcetera, etcetera. It's the same thing that is happening with eye tracking. It is the same thing which is happening in the automotive industry where you have used of facing cameras to

detected drowsiness. Except it's the same thing that is happening in in v R where a lot of people are are looking into eye tracking cameras or user facing cameras in in in, in the in the in the headset. So it's a it's a general trend basically. And I had a chance to experience this and many years ago. In fact, I think it was a booth at c E S and I think it was a Toby booth if I'm not mistaken, where I sat down and got a general demo of the eye tracking technology. At the time,

there was an application. It was a game essentially sort of like the old game of Asteroids, where you would focus on items on the screen, and wherever your focus was, that's where the quote unquote firing of the the asteroid destroying laser or whatever it was would focus. And it was a great little demo showing the promise of it. I imagine that as all technologies do, this has evolved to become much more accurate and precise over time. Yes, so that's obviously and one of our core challenges and

tusks is to improve the accuracy and precision. But it's also to make it work consistent live for the entire population in all different conditions. That is what it is hard. With an eye track there, making an eye track of the work for one person is prettycy, but making it one an eye tracker the works the same way for everyone is pretty hard. You can compare that with voice recognition.

I mean, making a voice recognition system that that recognizes your voice is pretty okay, pretty easy, But making to do with it for the entire human race in all different conditions, with all different background sounds, that's pretty hard. Um. So that's that's the challenge, and that's that's what we

are ultimately they're leading the world on. So you're you're looking at the separating all the signal from the noise making sure that on any I mean, I sit there and I just think about all the different cases where I'm using my computer. I mean, even if it were just a laptop implementation, keeping all the others aside just the laptop alone. There are times where I'm at my desk, there are times where I'm sitting at a couch or

in another environment. They're different lighting conditions, they're different angles. Um. I imagine that that for most of these there's probably some sort of UH training or orientation UH segment where it allows you to to at least set that baseline for eye tracking, for to make sure that you know you're getting a good response early on. But yes, as you point out, if you're rolling out of technology that's going to go to a wide consumer market, then that

challenges is non trivial. You have something that needs to work out of the box for UH an enormous really an endless variety of people in and an almost equally endless variety of situations. So um, these days you've mentioned some of the tools. One of the things I was really interested in was this user interface with computers specifically. But I mean, obviously all the differ friend. Uses of

eye tracking are are fascinating to me. Um. One of the things I was really kind of interested in is the implementation in the sense of how it works for uh in the pro gaming industry. We've been looking at that recently, and UM, I think it's really cool to have a new tool to analyze the way people who can play at a professional level. How you know where their attention is and is it different from people who play at a regular gaming level, like someone like me.

I consider myself a bullet sponge when it comes to games. I'm I'm the one that people practice on. Yeah, no, I'm great at doing that. I'm I'm if you have if you have a healer on your team who needs practice, I'm the guy who's gonna be giving that guy lots of work. But uh, I'm sarious. How have you seen this rolled out and implemented? Have Have there been any things in that implementation that have surprised you or any

interesting stories through that process? I mean this, this entire story is based on what we do in in in what we have done many many years in various business units, and it's based on the notion that what you look at is a good approximation of what you think because that is your intention. So the entire concept is pretty straightforward in that. And when you when you think about that, you can if I mean a pro gamer, where that

person spends his or her attention is extremely precious. I mean, they have so many different impressions coming at them at the same time, and they need to select and need to be very good at making sure they perceived the right things, making sure they spend their time on the right things, otherwise they they will get killed. Um so, and the concept is pretty much going to link to that. So, for example, a pro gamer may know that they need

to regularly check check the minima. You know, how many how many times a minute does a pro game or check the minimap. You're going to see them continuously going down there and checking what's happening, while a less you know, a less good gamer may focus too much on the game. You can you can you can check that out and see how many times are you're actually checking minima or you can see pro gamers are also before he's attacking,

checking the enemy items. You know, you know, what type of skills or what type of weapons or what type of skills does does the opponent have right now, you can see that a less less good game it may not do that. Yeah, you can also go into different players player styles. You know, when you are a certain player style, you may have a different games pattern than you are another player style because you need you need to do that differently. Or you can see things like

keyboard combination. I mean, you know, as as a really good game and you know you need to learn all your keyboard combinations by heart. You need to do that, um, and that's gonna be and by doing that, you can spend all your time focusing on the focusing on the screen. Well, less good gamers they will not be they will not know all the keyboard combinations. They will be focusing off screen for a certain percentage of the time. You can

notice that. So you can kind of say, hey, you need to practice your keyboard combinations because your off screen thirty example of time or trying to perstand the temperation on whatever it is you know. Um, so it's it's it's things like that that you can notice that can make you become a better gamer. If you study this is my favorite gamer, this is how they behave I see how they are playing and what they're intent is,

then you can learn from that. You may not have the same players toty hole, but you can learn from that and practice on certain certain events. Yeah, it might even just teach you what parts of the game you don't need to worry about because those are purely environmental and have no no direct impact on whether or not there's an opponent, and therefore you don't waste time looking

at something that is not a threat exactly. And you can see that just in the in the game where somebody is playing without tracking, they can turn on what we call a gaze overlay, and then where they're looking

you will get the kind of transparent blob. If the streamer wants you to see that, so they can turn it on, and then you, as as a viewer, get the much richer experience because you can suddenly understand that, all right, I know not only where the party, where the where the pro is clicking, but I also know where he's spending his attention, so I kind of can anticipate his next move because I can see a little bit how he thinks interested that, and that will make

you anticipate his next move and make you understand his gameplay better and therefore you can always game and therefore learn more. So just by looking how they scan off the environment, you can kind of follow how the pro is, how the pro is perceiving and playing the game, which is incredibly useful for us as a as a as a as a game that want improved. Yeah, this kind

of as a parallel. This reminds me a lot actually of poker tournaments that are televised, where you get a chance to see what the cards are of any particular player, which then lets you understand more about the decisions the players are making as the game unfolds. This is kind of the equivalent to that you're not just watching the game happen. You're actually seeing how the person playing it is.

You know what what they are perceiving and what is important to them, and that's why they make the decisions they make. So now you're not just seeing their decisions, but you see why those decisions are happening exactly. I mean, you you can take it to the to the you know,

real world poker world. If you're sitting around the you know, really fiscal table, then you really good poker players well will of course be looking at their cards, but they would probably spend a very large portion of the time watching their opponents. If you had eye tracking on them, you could actually see what are they focusing on, what parts of the player are they focusing on, and when you know and how do they know? So you can see what is it that they're focusing on, and what

are the interesting areas. As a non pro gamer, you would probably learn immensely by knowing, all right, I should not focus so much on my own cards, I should focus on new opponents x percent at the time. And then this is what the pros are actually looking at. He look, he's lick, he's looking at if player he is looking at player B and seeing if plays these blinking and if he's blinking, he's kind of folding or doubling. Yeah, so because because because that is the queue. Yeah, that's

and it's you know, that's incredible. It's it's blowing my mind to even think about that possibility. Because we know that people who or form at that at that professional level, at that level that is above like sometimes a huge leap above what we would consider a good player. There are these these components and people always talk about tells

and they talk about the psychology of the game. But that's stuff that is really difficult to learn unless you are pouring hundreds of hours of experience in to be able to see something like this, where you can get that direct feedback of oh, well, I understand conceptually what they're doing. Now I can actually see how they are doing it. Even though I would argue it would still take you many hours to to get a level of proficiency there, you would at least have a place to start. Yeah, exactly.

But you can take this also too, I mean, can take this to the real sports world. We have race Formula one race car drivers were NASCAR drivers who are using eye tracking in their helmets in order to see rick chord where they are in practice now, where they are looking and when they take the turns, because one of the most important things going to take a turn is to have the right kind of direction of your right,

right kind of focus points. And they're kind of learning from the really good drivers to the less good drivers. Where is the focused point in the term. So because they can see right the good drivers are behaving like this, this is their focused point. You know you should change. You're focusing on their own directions. So it's kind of it's it's in the physical world as well. Two. I mean you can see the poker examples very easy, and then you can obviously see the the the the the

the the computer game example as well. So this is a two way street, right, We're talking about in one side of things having a new approach, relatively new, I mean, eye tracking has been around for a while, but to see the technology mature to a point where it's rolling out more more consistently. In consumer market stuff, we see a new user interface approach. But on the other side, we're seeing a way of learning more about ourselves. And

so it's really cool. It's technology that you can take advantage of as a developer to create a new way to interact with a product or whether it's hardware or software. And then on the flip side, from a psychology standpoint, you can learn more about human behavior. You can learn more about human behavior and extremes such as the extreme

level of performance with professional gamers and that kind of thing. Um. So, to me, this is an amazing technology, not just for the potential of what I can do with it, but what ultimately we can learn about ourselves and thus design even better stuff another generation or two generations down the line, Yeah, I'm out in just a computer developer, if they could see, if they take a sample of a thousand gamers and users of the game, and they would know on what objects,

what what are these guys are in girls seeing in the user interface? And what are they not seeing? Alright, they're not noticing I mean, they're not clicking on this thing. They see it, but they turn away because because they're not interested in it, or they're just missing this thing. You know, how what what cues can you give it to you? So if you know what they have, what they have seen, you don't need to have a second reminder of some of something that somebody has seen, seen,

seen already. So you can improve the user interface immensely obviously by the stuff of right. And of course today now that we're in a world where people have persistent Internet connections, it allows you to roll out uh tweaks. You can roll out versions and patches on a very

frequent basis. This allows for incredible flexibility if you are trying to create something that is software based and you are getting real time feedback due to the eye tracking data about what is and isn't working, or what is or isn't necessarily considered were the a focus and either redesign it so that something that you think is important can have a better sense of prominence in the eyes of the people using it, or you just eliminate it.

If no one's looking there and it's taking up landscape, you might just get rid of that entirely, if no one really is using it. It's to me, it's like another leap forward where we saw advertising go once the

Internet became a real entity. Because of course, in the old days, advertising was limited to print and radio and television uh campaigns, and you had a sense of how well your ad campaign was going based upon sales, but it was really hard to nail down whether or not that was actual causation, right, because maybe it was because the ad campaign, or maybe there was some incredible sale

that really help boost sales. But then the Internet comes along and that gives a much more direct feedback loop of what ads are are encouraging people to click through and therefore experience it on a deeper level. This is like that, except it can apply to everything, not just advertising, but everything. So that's that's what I mean when when I say eye tracking gives understanding of your intent and how important is attempt to any any any computing device.

Intent is is probably the most important courage in that you have. You know, you're you're you're kind of intent and you're interesting. Your attention, I mean that is the most the attention of the of humans is the most pressured pressure pressures, pressure resource you have. Right, but then it jumps down that, right, you can never share eye tracking data the user doesn't want it show. I mean, it's it's the user that controls it, so so there's

no kind of possibilities to spy. It's like you as a user, you control if you want to give this to a corporation or if you don't basically like anything else. That's super super important. Obviously, absolutely yes you wouldn't if you're volunteering it, that's one thing. But if you just found out, oh gosh, Facebook implemented this technology and now they know everywhere I look, and now my feed is being filled with puppies, and I like puppies, but I

really would like other things besides puppies. Uh. I mean, obviously we see that sort of analysis in all areas of technology that are going on, just based upon our actual physical behaviors. Are choices made, like whether we're following one link versus another link. But I think a lot of people would really uh, if it weren't something that they were volunteering, they would really have a negative reaction to that, to think, oh, well, now it's gone beyond

what I'm choosing to do. Now it's going to what I'm I'm actually looking at. And uh, I think a lot of people would find that creepy if it weren't a volunteer, voluntary thing. So I agree with you entirely that that that's an important step for this to be a technology that people embrace as opposed to reject. Yeah, and that's going we put hard viruses with the guardians of the end users privacy or when you can only

read it if you're if you're loving right. So what I want to ask you what your personal favorite implementation of eye tracking technology has been up to this point. It's it's a very good question. It's also very hard question because as we have discussed, his knowing the intent of humans is so powerful, So you can you can drew up so many different example and implementations. But if I, if I, if I take a few, um we have in our business unit Tobo Dona woks. We have so

many users. So when their mothers and fathers who come to us and say, hey, my my son or my daughter has for the first time ever communicated to me, mom, I love you, or has for the first time drawn a drawing on on on anything, but this time on the computer with with his or her eyes, I mean that that goes beyond anything when you get that type of really really kind of strong value, because because a person who has previously not been able to communicate can

now communicate, I mean that that's kind of that's outside anything. Yeah, that's transformative. I mean that's that's the sort of story that is incredibly inspiring, and it does show that this technology has an incredible power, and it can not only does have incredible power, it has incredible power to empower people who previously had very little chance of having that

sort of human connection. And until you hear those stories, you don't even realize how incredibly valuable that can be and how much those of us who haven't had to experience that take it for granted. But then you start thinking about it and you realize this sort of connection that is possible due to technology like this and you really, I mean, it really does open up your eyes on didn't mean to go with that particular metaphor, but that it really it really does does hammer home how how

incredibly impactful this sort of technology can be. The other the other if I take one more example, more from the consumer world. You know, I don't think it's hard to tromp that type of but take it from the consumer world. If I think about a VR headset or or an a R or a R air headset, it's what what is happening is we are actually seeing that we can take a three step interaction and make it into two step interaction and take away one entire step

into every single interaction. So interview had set us of today the method to interact with something is one, you look at it too, you turn your head or you point your hand controller to it. Three, you push a button to pick it up or you know, whatever you want to do with it. So it's kind of one look to turn your head or move your hand control to the point, three press a button. You know. Their variations could be my voice as well, but there so

that's a three step process. With eye tracking, you reduced that to two step process, which is basically one you look and an object to your press. That's fantastic, and it creates a deeper sense of immersion because you're no longer again you're removing another one of those layers that the person using it had to go through previously. Every time one of those layers gets removed, then you have this this deeper immersing of the sense that creates a

more convincing experience overall. Yeah, and when you think about that, what when you think about what the entire touch revolution, what that really is? You know, you know I when you've given a touch phone to you, you know, your first experience with a touch phone with and and and the you know the and the what that revolution is is like one you look to your touch. That is

what you do on the touch phone. It's a two step in the direction one too, just like it is in the order with eye tracking, one to press about them while it's coming from the PC world, where the interaction is one look to drag with your mouth or your touch pad three click. So the revolution you're going through with eye tracking is is exactly the same in terms of taking one step away and just having it

one look to action, one look to action. So it's I would argue, it's just as powerful as that touch revolution. And when you think, when you think about it further, why is this so much more intuitive? It is because that is how we have learned to interact, interact even since we were a kid. You know, when you, as a kid wanted to learn we can interact, to interact with something. You have something on the table in front of you, it's one look to pick up to explore.

You know, when you want to turn you on your dishwasher, I mean one look at the knob to turn. When you want to shake somebody hand, look at the one look at the hand to shake the hand. You know, it's it's a two step interaction. That is what you have learned as a kid to explore, one look to pick up and and we're replicating that. And that's why

it feels so intuitive. And I can tell you anybody that we demo a VR or or are application to they kind of they get it instantly, like, oh yeah, this is more intuitive because it's simple as that we're just replicating what you do in the red world, which is which is kind of actually, if you wanted to virtual reality, it's it's supposedly pretty important to replicate your actress in the real world, right, yes, I mean if it If it doesn't, then it takes you out of that.

You're aware that you are in an artificial experience. And uh And obviously the goal is to reduce that awareness as much as possible so that you can really have that the experience of the developer intended when he or she started to design it. Uh And and ideally, at least for most VR experiences, that's to strip away all

the rest of the awareness in augmented reality. Obviously it's meant to augment and experience in the real world, but even then, you want to you want to reduce the load on the user of having to think, oh, well, if I want to accomplish goal X, I have to go through these steps in order to actually do that.

Whereas in the real world, if you were looking at an actual physical rep of what you were seeing in the argumented world, you wouldn't have to go through that because that's just not the way the real world world works. Imagine imagine, imagine a fast paced VR game or a

R game, it doesn't matter. I mean, you would be running down a pitch in a in a football match, and you want to pass the ball to somebody who's running full speed in the other direction, and then you how do you do that with the current controls in the R I mean you're running full speed your yourself, and then you should either point your head to kind of to to point to the other, to the other, to your teammates running in the other direction, and do

that continuously to tell the computer where you want to tell your head that way you want to pass the ball, or you should running full speed one direction. You should point your hand to tell the via heads that I want to pass the ball to this guy. That's super hard. I mean, that's just gonna It's just impossible. You can try to do it yourself in your any room if you want. But or you imagine the interaction. You're running down, you know, in one direction, you look at the guy

running in the other direction. You can do that. You can follow that person extremely easily with the rice and then you just press a button then post that a guy. That's that's just so much easier, and it takes down this barrier. It takes away one step and it will enable game developers to make Vio games more immersive and also more frost pace, because otherwise you need to kind of slow down the pace in order to make people

hit the right person. And not only that, I would add that it adds in an element that you see in the real world all the time, which is misdirection, purposeful misdirection. If you're playing on a team and you want to pass to a teammate, you don't want to

telegraph that to the other team. So so being able to use your attention to direct the ball in the in the right direction without having to turn your head and make it obvious, oh well, he's about to pass to his teammate on his left, then it makes it more challenging for the opposing team to anticipate what your move is. Yeah, or more I mean more like personal connecting interaction. I mean you're sitting around the table, you're

talking to set up people. Imagine you have two people in front of you now, and then you start talking to one of them. You would be looking at that person, um, but you would be looking at that person with your eyes. You would not turn your forehead to that person. So the v R A R head said would not know that you're you weren't talking to person A. It may you may have your head the point still the point

that that person be. So then the via head said, would it would the wrong person would not or blink or or or or smile at you when you when you smile back at them, because they you don't know what you're what, what you're interesting. You only know what you're interested in, what you're interested in if you know where you're looking, because your forehead is just an approximation of your interest, and it's always it's going to be wrong in in certain number of cases. Oh sure. Yeah.

So if you were in a game that wire's interaction with uh PC controlled characters, and you're trying desperately to uh to align yourself with one character, and there's a secondary character there, and you accidentally swear fealty to the bad guy, Suddenly the whole game has changed, yeah, or just feels less immersive because the people you look at their stone face, they don't want to spot at them because they don't understand they're looking at them because you didn't.

You didn't do you didn't turn your forehead to them to indicate that you actually want to speak to them, which is I'm not sure you don't do that really well and I could see this also just being useful in a virtual meeting space where you're just having a like just a not a game element at all, just a literally a meeting where you have multiple people logging

in through various devices. Being able to see where anyone's gaze is is incredibly useful because if you're addressing questions to somebody, you don't have to It's it's kind of like inter acting with a personal digital assistant that's voice activated today where uh, you know you. You use a typically a name that indicates this is the alert to listen in and then to respond. But it's not the

way we humans tend to talk to one another. If I were having a conversation with you, and I started every sense with Oscar, what do you think about after about after about five or six sentences, it feels weird. So you've got it. You've got it. Can I hire you? No? But that's that's it. You can I take one more use case, just absolutely please do or another I mean just to take it in a different direction or tangent. It's it also can also affect the actual core rendering

of the graphics. So I mean today in VR or in PC, I mean, we're all creating to create beautiful games with with as high resolution as possible, but that takes a high toll on on the on the GPU mhm um. But actually the human eyes only perceiving what you look at in high resolution where you look at it,

the rest is kind of blurring. So that means you can do something called phobid rendering, that is that you only render in high resolution in the area you look and in low resolution in the prefery, and then you look at a different point, and then it switches very fast to render in high resolution where you look at that point and in lower resolution and periphery. As a user, you cannot notice because this is all happening too fast. You you're not noticing that the entire world is rendered

in lower resolution in the prefery. But you save thirty or two on the GP capacity excellent. So then that also means that you're not overheating your machine. It means that devices that have say the latest and greatest in in graphics hardware, that's going to remain relevant longer that it would be like the life cycle of these things is notoriously fairly short. If you want to be on

the cutting edge of capability. But if you're using a strategy such as this that reduces that load, then you've extended the life, the useful lifespan of those uh, those those components by quite a bit because you don't you're not maxing out as quickly as you would in the in the real world. I mean I that was one of the reasons why back in the nineties gaming computers got to the first they became a thing, and then they became a thing that I just could not keep

up with. And it was very frustrating for me to think that every six months my machine was out of date enough where if I went out and purchased a new game six months after I bought a machine, or I bought a graphics card, I could not run it at the settings I wanted to because it would be too strong, to too big a strain on my components.

I would have to actually upgrade my hardware. So if you're able to solve that through software where you're you're able to reduce that load, to me, that's phenomenal because it takes me out of that massochistic relationship I had have. You can be also be a little bit less of a bullet sponsor. Right, Oh, yes, that would also, that would be lovely. I mean, let's not dream too big here, Oscar. Let's let's keep our expectations realistic. No, but you can.

You can use it obviously to keep your hardware longer or to reduce your cost of your hardwork. Or you can use it to get a better immersion by being able to get your computer to handle high resolution graphics. Or you can get your computer to um to run

at the higher refresh rate or higher FPS. So you can use it, you know, however you want either to save cost or prolong your prolong your lifetime, or or two to to to run the game in a in in higher solution and get more Marcy, that's fantastic, Oscar. Thank you so much for joining me on this podcast. This has been a fascinating conversation. I very much appreciate your time. Alright, when we come back, we'll talk a bit more about eye tracking technology and where it's going.

But first let's take another quick break to thank our sponsor. As Mr Werner pointed out, the data we generate through our attention is valuable, and it also poses a clear threat to our privacy if it is handled poorly. Now, I appreciate that Toby's policy is to only use data with the permission of the user, because you can easily imagine scenarios which could be compromising if you did not

realize your eye movements were being tracked. So, for example, let's say they're going in for a job interview for a big tech company, and you're qualified, and you are knowledgeable in your field. You are a hard worker and you really really want this job, but you're also a bit nervous about going in and interviewing. That's to be expected, and because you're nervous, you know, you're looking around a lot.

You don't really want to hold anyone's eye contact for too long because you don't want to come across as too intense or anything. So you know, you're you're just trying to feel your way through this experience as best you can. And unbeknownst to you, but beknownst to us, the people meaning with you are also using a system with eye tracking technology to observe you throughout the interview process, and the system is analyzing you as you respond to

questions and converse with others. It's watching your eye movements and tracking that against various behavior patterns and drawing some conclusions, and maybe some software is generating a report that assigns metrics to fuzzy categories just assertiveness or honesty, and maybe even though you really are the best candidate for the job, the system tells the hiring manager that you're nothing more

than a shifty peat and they shouldn't hire you. Now that kind of sounds silly the way I'm putting it, but it's certainly something that could potentially happen, and the technology already exists. Really, it just means that you have to create the software, which largely would mean making sure that your software is at least on the surface, dependent upon reasonable psychology. Right. You can't just say I think

that if you look up, you are lying. That's one of those little bits of folk wisdom that actually isn't true. So you would want to make sure that your software is based off of science, or at least appears to be based off of science. I mean, if you're not being very ethical, you could sell your software package no matter what. But it's easy to imagine that sort of scenario actually happening. That a company be that assertive or aggressive and its hiring strategies to incorporate that level of

screening when they're looking at potential employees. But it's not really fair and it certainly seems invasive. Now that's the hypothetical. It's not something that I'm actually referring to that's happened in the real world to my knowledge, but it's the sort of stuff we don't want to see. It places people at a disadvantage, and it could also be used to create opportunities for abuse in many ways, which again is why that policy of only collecting information with the

express permission of the user is important. You wouldn't want to come back at somebody and say, hey, I noticed that you were spending an awful lot of time looking at this one person in the room. Uh. It makes us uncomfortable because it appears that you have some sort of attraction to or maybe animosity towards this person. All of that seems like it could go south in a hurry, and it be reflective of something that is not actually real.

Because while our eyes are an indicator of where our attention happens to be, it's not always the case that what our eyes are pointing at is what we're actually focusing on. It's most of the time that's the case, especially when we're being very active with our attention, but it's not always the case. So this is a complicated issue. I think eye tracking tech has far too much potential to do great things for us to be too wary of it. I don't think we need to back off

of it. I don't think we need to abandon it. I don't think we need to label it as irresponsible or dangerous technology. We just have to make sure we hold organizations, companies, researchers accountable for responsible implementations of the tech. But these applications really could transform our world, particularly for people who have difficulty interacting with others or with their environment if they don't have that kind of technology at

their disposal. So, like Mr Werner said, a technology that helps someone communicate after it appeared that all such ability had been lost is an incredibly powerful story, and it's hard to think of something more impactful than that. On the research side, scientists are using eye tracking to learn

more about cognitive development. There's a researcher named Alex Willard who is working on her PhD and Maternal infant directed speech at the University of Newcastle and Hunter Medical Research Institute in New South Wales, and she's using eye tracking technology to study how babies develop cognitive skills. She's learning how babies recognize and solve problems. And I didn't even know babies could do that. I haven't seen Baby Boss though,

so maybe it's all explained in that movie. Anyway. The study involves babies watching animations on greens and it tracks where their eyes go. So, for example, let's say you've got a cute cartoon bunny hopping around on the screen, and the baby notices the bunny and the baby's eyes are tracking where the bunny is going on the screen, and then the cartoon bunny hops behind a cartoon bush. Now at that point does the baby's focus continue along the bunny's pathway to the other side of the bush,

anticipating the bunny's reappearance, And if not, why not? Now? Early results in the research project, and this project is scheduled to last for five years. It's a long term project to make sure that any findings are things that that can be backed up with replication. Some of the early research has suggested that babies who are less distractable when focusing on a subject tend to be better at problems solving later on, which makes sense intuitively, you would

imagine that to be the case. But this approach could help parents detect potential challenges early on and address them and give their children the best chance of overcoming those challenges as they grow older. So you might see that certain cognitive development might be running a little bit behind average from you know, the mean, and you might say, well,

what can we do to address that? Are there things we can do as parents to help encourage our child's cognitive development in these areas and thus give your kid the best chance for success. It's kind of cool, I think, and it gives a lot more information and power to parents, stuff that people didn't have at their disposal without tools like this eye tracking tech that can back up those observations. Now,

another study sounds ominous upon first glance. I was reading headlines and this one jumped out at me and it made me pause. It was the National Institute of Health spending four thousand, two hundred twenty dollars on eye tracking technology to analyze the eye movements of Latino customers at grocery stores. That was essentially the headline which made me say, what is going on here that sounds minority report esque, but upon further observation further investigation, it's not nearly that scary.

It's actually kind of interesting. In this case. This is a study at San Diego State University that has the goal of finding ways to fight obesity, and the system is meant to provide data about the decisions that overweight or obese people are making as they shop for groceries, and it's in an effort to get insight into what is causing them to make those decisions. Are there specific things in the grocery store environment that are triggering those decisions.

Can adjustments to the environment help guide customers into making more healthy choices? And the reason the study is looking at the Latino population in particular is that, according to the researchers, as a demographic, Latinos are disproportionately affected by obesity. They also shop more frequently than other populations do, and they tend to go shopping with children more frequently than

other demographics do. So the hope is that by using the information from this study, we can come up with strategies that help people make better choices and have a healthier life, and also provide a healthier life for their children. In the consumer world, companies like Toyota are using eye tracking technology to measure how effective their show rooms are drawing attention to the features that they want to communicate

to potential buyers. There might be something on a new Toyota model that the company really wants people to pay attention to, and they use the eye tracking technology to see if it's working, and if it's not working, can they adjust their approach to make it more attractive. And inside vehicles, we're seeing more eye tracking technology, both as a safety measure in the case of systems that monitor a driver's wakefulness or attention, and as a user interface.

There's a Canadian research team that's using eye tracking technology to look at the potential for driver distractions. So it's monitoring a driver so that if the driver takes his or her attention away from the road, Let's say they are checking the various instrumentation panels, or maybe they're looking at their phone trying to read a respond to a text. The system the car would actually become aware of this.

Maybe it would start to engage some driver assist features, maybe even taking over control of the car briefly, or it might send an alert to the driver saying, Hey, your attention is wandering away from the road. You really need to pay more mind to the task at hand. On the user interface side, Porsche recently revealed an electric car concept called the Mission E. I should really get Scott Benjamin in here to talk about this concept car.

It's kind of neat. It's an electronic card. It's supposed to be a sports car that could compete with the Tesla sports Car electronic sports car, and these would include eye tracking type control systems for the in dashboard systems as well as the the entertainment systems, the various instrumentation panels. It would be able to detect what you're looking at

and give you information based upon that. And I imagine we'll see even more of this technology built into self driving vehicles in the future, when the cars we get into become interactive environments that allow us to do all sorts of things we couldn't do as human drivers. And again, this is just scratching the surface. There are so many more potential applications and I'm sure I'm going to visit this topic again in future episodes to explore some of them.

If you can't tell I'm really interested in this subject because it opens up new possibilities to interact with technology, will similtaneously giving us the chance to learn more about ourselves, and I kind of dig that. Before I sign off, I have an exciting announcement to make. I have a new podcast launching. It's called tech Stuff Daily, and as the name applies, it will be all about technology and

it's also going to publish every day. That's that's what the daily part means every day Monday through Friday, and each episode will be between four and six minutes long, give or take, and we'll explore a tech topic that's been rambling around in the news cycle. This is not replacing Text Stuff. That show is still going to be publishing twice a week as always. It's going to exist side by side, but now you'll get even more tech

goodness by subscribing to tech Stuff Daily. Now, in my next episode, i'll chat a bit more about using eye tracking technology specifically in the realm of e sports, as well as a discussion about the sports in general. I look forward to chatting with you guys about it, and if you have suggestions for future episodes of Text Stuff, please let me know you can write me at tech Stuff at how stuff works dot com, or drop me a line on Facebook or Twitter. The show's handle is

tech Stuff h SW. Remember I record new episodes every Wednesday and Friday, and you can watch me record them live at twitch dot tv slash tech Stuff. Just visit that you r L. You'll find the schedule there and I'll talk to you again really soon for more on this and thousands of other topics because it staff works dot com.