How Facial Recognition Technology Works

Dave says, Hi, guys, love the show. I've been wondering how facial recognition technology works. How does it know what a face is and whose face it is? What are some of its uses fun and practical? Are there dangers and controversies around this technology? What's in the future for this stuff? Thanks for all your fascinating discussions. Have a good one, and I should point out also this email other people have asked us about facial recognition technology, so

Dave's was the first one that I came across. It dated from February of two thousand and ten. It is currently as we're recording this October of two thousand and ten Dave, I'm sorry, we have lots of other topics just like that, so we got plenty of stuff. Keep letting us know what you want to hear. Yeah. So, but a lot of people have asked about face recognition technology, and we thought, I thought it would start with kind of just a brief discussion about face detection technology because that,

you know, you really you build upon that. And I think a lot of us have used digital cameras at this point to have some face detection technology built into them. Yes, it's it's not uncommon, No, not at all. Now here's the thing about recognizing and detecting faces. People are really good at that. Yeah, well most people are, right, most

people are. There are exceptions, of course, but the the average person who doesn't have any uh problems with his or her site or have any problems within their brain that makes it difficult to detect and recognize faces they check, they usually see it right away, right, you know, you just look at a person, you see their face, you recognize that person if you've if you've seen this person before, more often than not, you'll recognize that person. Computers aren't

very good at this. Uh, It's one of the things that that really is a barrier in artificial intelligence is that humans are very good at detecting and recognizing patterns and and keeping that information stored so that they recognize it when they encounter it another time. Yes, in fact,

we're so good at it, we sometimes make mistakes. This is this is often comes out and paradolia, which is where you you see patterns in stuff that there's no actual pattern there, Like you look up in the clouds and you see, hey, that just looks like my buddy Joe's face up there. Well, that's that's our brain creating a pattern where there's not really a pattern there, right.

But computers are not traditionally very good at this. It's actually a big computing problem and problem in the sense of how do you teach a computer to recognize patterns? Face detection and face recognition kind of that that's right up there with that problem is how do you teach a computer what is a face? Well, that's a little bit tricky. Of course. This is done with software and UM it relies on on algorithms, which are you know, a sense of instructions basically for for computers or or

anything running a kind of software like this. UM and what the what the researchers and engineers have had to do is install a layer of software that enables these devices to uh you know, they had to basically teach it what I mean, not literally they we're not talking artificial intelligence, but they had to basically teach it what is a face? Right, So often software software may or may not be the right term, depending upon which device you're using. Firmware maybe because often it is hard coded

directly onto a chip. But yeah, it's the same same principle, right, It's it's not it's not hardwired onto the chip itself. It's it's a program that exists. There. I apologize for my mistake. No, no, no, there's plenty of stuff out there where it is a layer of software. It's not firmware. It all depends on and we we've talked about those definitions being fuzzy anyway, right, so I'm just trying to Yeah, so anyway, fuzzy firmware. That'd be a great name for

a band. That's my Devo cover band name. So so yeah, the generally what the this firmware or software, what this program is looking for are the basic identifiers of the average human face, which would be, uh, your eyes knows your your ears, your chin kind of the outline, and when it recognizes that basic pattern, the the software identifies

that as a face. So if you hold a digital camera with face identification software on it or or that feature is enabled, if it sees a pattern that looks like ears, eyes, nose, and chin, it's going to immediately assume that that's a face. Which sometimes it can be funny, like you can sometimes get face recognition software to recognize a face on a on a like a picture, like it's not even a person, You're like a mural on

a building. In fact, I remember Google street View. They use an algorithm that's essentially this, you know, it looks for those features in order to blur out faces. That's part of the privacy uh stance that Google takes is you know, people were objecting to having their their pictures on Google street View. So what Google did was they create this algorithm that looks for the human face and then it applies a blurring layer over it so that

you can't tell who that is. Right. Well, I've seen that work on things that were not human beings, mostly on things like like billboards or there was one that was a mural that was on the side of a building and the the face on the mural had been blurred out automatically by Google street View. Well it is a face. Yeah, it's just an actual three dimensional face. There's a good chance that mural did not object to

having its privacy, uh review, you know, violated. But anyway, so it's looking for that basic set and that's just your your your very basic face detection. It it it it says, it looks for this this pattern of images and says this is what a human face is. Now when you go to face recognition, where it's going beyond detecting a face, it's actually recognizing a face and and setting that to an identity, we get a little more complex, actually a lot more complex, true enough. Um, yeah, it

does rely on those facial landmarks. Yes, you know, your eyes and nose chin, the depth of depth of your eye sockets, the length of your nose, wid your nose, yeah, nostrils, everything like that can be part of this. And and for the for the camera in this case, it's recognizing a face print sort of like your thumb print or fingerprint. Um. And so once it it can keep a record of what a particular face looks like, um, then it can

can you know that? That's I think you would probably call that the first step and being able to identify a particular face. So you take a person's face and you look at these different measurements. It might be the distance between the eyes. Uh, it can be things like the width of the eyes themselves, um, the where the ears are in relation to the the head as a whole, um, the jawline, all of these kind of features that you really want to focus on. Features that are not easily changeable, right.

You know, like hairstyle would be a bad measurement because you could easily change that, right, right. You want things that won't change over time, right, So you know it might be things like again the width of the forehead, that sort of stuff. These are all called nodal points, all right, And uh, we have a great article about facial recognition technology on how stuff Works dot com. And in that article you learned that that the human face

has around eighty nodal points. Now, not all facial recognition technology is going to rely on all e d of those in order to create a face print, but they'll rely on some combination of those nodal points and through the measurements will come up with a numeric value, which is the that's the equivalent of the face print. It's a numeric value that is unique to that person. More

or less, identical twins actually can have the same face print. Wow, yeah, it's it's so some facial recognition technology cannot discern between identical twins. There are other kinds that rely on even more specific data that can discern between the two. But a basic facial recognition camera, Um, if you had identical twins and they really were identical, yeah, like they didn't have one of them didn't have some sort of facial

feature that was remarkably different. Uh, this software might identify both as being the same person without without additional layers. And we'll get into that in a little bit. So you've got this face print, you create a database of face prints. Then when you take a picture of someone the again, the camera will measure the noal points on this person's face, compare it against the database and see

if there's a match. And it's probably I'm guessing we don't really go into a whole detail in our article, but I'm guessing it's kind of like fingerprints in a way, you look for a percentage of probability that this this particular face matches one that's in the database, because you got to remember, not all of these images are going to be exactly the same. Early facial recognition technology was

very limited. You had to have someone looking directly into the camera, and then you would have to have that same person look directly into the camera again later on and compare that to the database in order to find a match. If the person was looking a little to the left or to the right, the technology wasn't good enough to to compensate for that and to make a model of that person's face to really get the right measurements right. So, and you've got to think of all

the other factors that play into this. It's the lighting. If the lighting is bad, those early facial recognition technologies weren't very effective. Or if the person was at a different distance. The camera has to know how far away you are in order to make valid measurements for things like how far apart your eyes are. If the camera thinks you're fifteen ft away but you're really twelve feet away, those measurements are not going to be accurate, and it's

not going to find the right match in the database. Right, So this is this was definitely one of those things that was a big learning curve you had to be able to build. You had to develop the digital technology to detect distance and then accurately measure as many Noble points as possible and as little time as possible. And we're talking hundreds of a second here. Uh that that a chip is scanning a person's face and identifying those

nobal points. I mean, it takes no time at all for this to happen, but it took engineers time to develop that technology. And yeah, there there are some really fascinating technologies built into facial recognition, including you know, technologies such as a surface texture analysis. It's basically creating a a another not a face print, but a skin print. Where they are doing, um, the system is doing a mathematical analogy of sections of your skin if you come

before the camera. So um, if you had to say a birthmark or a mole or something, that would probably help it track down who you are because it's going to say, well, we know that this is you know in sector number seventeen. Um, you know there is this different coloration than there there would be in the rest of of the face, so um that that you know, they're very sophisticated in breaking them down into um, you know,

all kinds of mathematical UH constructs to enable this to work. Yeah, And the service texture analysis is that layer I was talking about earlier about how to identify between identical twins. Service texture analysis is actually the kind of of technology you want in order to do that, because it's looking much more closely at the texture of your skin. As the name would imply, so uh, even identical twins aren't going to have identical lines on their faces, you know, laugh, lines, wrinkles,

that kind of thing. There might be a freckle or a mole that's it's slightly different from one twin to the other. And this is the sort of technology that's going to pick up on that, as opposed to the basic facial recognition technology that might not it might be close enough between the two twins in order to identify that as the same person. But with this layer, it can make it more accurate. In fact, according to one company, UH,

it can, which is called Identics. UH. Surface texture analysis can increase the validity or the the reliability of a scan by around which is pretty significant. I mean, you know, you know, depending upon how accurate your starting point is. The early early facial recognition technology, even when it was working well, was not working that well. It was like around a sixty success rate. And uh, you think about that, it means of the time they get it wrong, and

that can be pretty serious when you consider that. A lot of the facial recognition technology that's out there is used in law enforcement practices. Yes, it's in order to identify people who you know, let's say that there's unknown suspect that is on the loose, may be used to try and identify someone like that, or it may even be used in a static system where if something happens within the view of the camera, the camera would be

able to to identify that person more quickly. You wouldn't have to just you know, stare at this picture and say, I wonder if this is you know, is this a suspect or you would have compared against the database of like mug shots or whatever, and um, that can be really useful in UM in urban environments that have lots and lots of cameras that they use for law enforcement purposes, places like London where there are so many cameras on

all the different street corners, but also in places like banks. Um, if somebody were unknown uh you know, unknown bank robber and they didn't where you know, some kind of facial obscuring gear. Now, I mean it would be easy enough to fool a bank camera, probably if you put on a fake mustache or something like that, because that's the kind of thing that's going to sort off the facial recognition. Yeah, there's some facial recognition technologies that are they're sophistical enough

to ignore things like facial hair. It'll look at the shape and size of your face and the relationship of say, like again the distance between your eyes, and it ignores things like facial hair because again, facial hair is one of those things it's easy to to grow or remove, right for some of us anyway, ladies, hopefully not for you. But the the most facial recognition technology will ignore that

kind of stuff. But yeah, if you're wearing something that's obscuring your face, that definitely will throw off facial has to and that's not gonna be able to recognize it because I can't see it. Um. So if you walk into the bank wearing a balla clava, the camera probably won't recognize you, but the security guard might have some issues, right, And this also leads us to the question about what sort of problems there can be. Well, clearly, privacy is

a big concern, right. I mean, if you get to the point where you have technology that can recognize faces, then you've got your your to the point where you have to worry about you being viewed wherever you happen to be and identified as being there. And I mean

that's a big problem. I mean even for people who, let's say that you are perfectly innocent, upright wonderful citizen and you've never done anything wrong, you still probably wouldn't necessarily want all these cameras everywhere identifying your your you wherever you happen to be. I mean, it's just it's

it's an invasion of privacy. And because of that, there have been some experiments with this sort of system in place in various public areas that ended up getting um canceled somewhere along the project because either the public was in an uproar about it, saying, hey, this violates our privacy and I'm not comfortable with any agency tracking me like this, or the reliability was low enough so that there were concerns of Hey, I could be at home asleep and someone who looks enough like me for the

facial recognition technology to think that that was me could commit a crime, and then I could be charged for it. And I mean, if the if the reliable reliability is low, like if it it wasn't that six, that's a legitimate concern. Why if you happen to be involved in one of those mistakes, like Jonathan, why did you break into Tiffany's and steal all these diamonds? And I'd be like, wait, what did I do? Now? When did that? I? I

am certain I did not do that. It does not say I ever went to Tiffany's on four square, So clearly that wasn't me because I check in everywhere, right. But there can it can be a lot of positive uses of course for facial recognition. Oh yeah, no, there there are plenty of really good ones. I mean not that those aren't positive, but I mean more fun let's

see uses. Of course. You know, with cameras being as sophisticated as they are, photos are geo tagged, and now you know the facial recognition software built into them, you can auto tag different photos as basically as you are, you know, importing them into your files. Yeah, that to me is absolutely amazing stuff. I was amazed back when cameras first started being able to detect faces. That to me was that to me was really really cool. I

was like, hey, awesome. And then you had the next step beyond detecting faces was detecting when someone was smiling, because then they were measuring the person's mouth right, and so if a person was smiling or making some sort of facial expression that was akin to smiling, because sometimes it was like more of a grimace. Uh, your camera would merely take the picture. There's some cameras that had it where it would it would be ready to take the photo and as soon as the person smiled, that's

when it would take the shot. So that way you would get the smile right. And then the next step beyond that is what you were talking about, where you would tag a photo. You take a picture of someone, you tag that that photo with the person's name, and then from that point forward, your camera compares the pictures you take against the people you've already tagged and says, oh wait, this is a person he's tagged already I'm

just gonna go ahead and throw the tag on there. Yeah, that's a good point because just as the manufacturers roly on the software engineers to build the technology in so that these devices can recognize faces, you have to teach it who is who? Um? Who is whom? I should say, um, so you know, it won't know that when I take a photo of Jonathan that is Jonathan until I tell it, you know, this is who this is, and then therefore after that it will take photos right well, I reckon

nize his face. It's Jonathan Strickland. And you know when I upload those embarrassing photos of him to Facebook, it will, you know, have all the impertinent information included properly, which is why every evening I have to sit down and untagged photos. Yeah, but you know, I was thinking of another uh technology facial recognition application I should say, Um, that is also pretty fun, which is uh Microsoft's connect. Ah. Yes,

that's a good point. Yeah, connect is of course, that's the motion detecting right always, because that's what the that's that's what the project was called before it hit the public. How you associate a name with something and then it's sort of hard to unassociated with that. I still call segways ginger. That's it. That's going way back. So anyway, Yeah, Connect has facial recognition built into it, and it's the implementation.

Like you were saying, pullet is a great idea weather weather or not it works well, I can't say because I haven't used Connect yet, but the I love the concept that when you step in front of your your entertainment center, the camera in the Connect peripheral looks at you and then analyzes your face. Does this this process that we're talking about where it measures, takes these measurements very quickly compares that to the information and database and

then identifies you. So if you have previously set up and a Connect account, let's say with your Xbox, it knows, Hey, this is Jonathan. Jonathan likes to play at this particular skill level. Jonathan likes these particular games. Um, I'm going to present this this block of information and features and games to Jonathan because we already know what his preferences are. Then when someone else, let's say, Chris, steps in front of Connect, it will identify Chris and say, oh, well

that's Chris. Chris like some of the games Jonathan likes, but he also likes these other games, and he prefers these kind of movies to the movies that Jonathan likes, like Chris prefers the three movies they've seen before to the vast database of movies that Jonathan has seen. So I'm just going to show them these three movies because there's no point in trying to get him to watch anything else. And then, um, but that's the thing is that it'll it'll tailor the experience to the person based

upon that person's profile. Now again, like Plett was saying, you have to create a profile. You have to tell connect Hey, this is who I am. Whenever you see this face, this is the profile you should use. Now, I'm sorry you're going to say something, Well, I was just if you were going to extrapolate from that I thought of I just thought of another application similar to that that Apparently, as I look it up quickly, here, Um, other people have our way ahead of me, and doesn't

surprise me in the least. Um, smart homes can do the same thing, because I know that. I remember reading a long time ago that when Bill Gates had built his you know, massive home with the Microsoft technology state of the art, and it would tell as soon as you walked into a room, uh, oh, well, you know this is Bill. He likes the temperature at you know, seventy two degrees, he likes this kind of music, and it would automatically like you could walk around the house

and and I remember reading this. I don't know if it's still true or not, but they could follow you with he's like, oh, well, you know, we'll turn on the music in this room, we'll turn it off in that room because he's no longer there, because you know, we know where he is. But you would have to as I remember correctly, Um, like I said, I just thought of this on the fly and didn't research it. But as I remember correctly, it relied on some kind

of r F I D technology. You had to be wearing something like like a little name tag that you wore, but you could you could use facial recognition technology instead, sure, and not have to carry anything with you. And then you wouldn't have to go, oh, well, you know, dude, I left my card in the in the laundry room, and now I'm in the living room and I really don't feel like it not right. You could tell where Pallette was based upon the game show and commercial music

that you heard throughout the place. Oh nice, thank you, welcome much. For me, it would be musicals. So I mean, I'm gonna go ahead and say, like, who's doing Fossey. That's gotta be Jonathan. But yeah, I mean it would be. It would be highly useful. And there are people, as I you know, run a quick search on the search engine, that are already ahead of me on this and have you know, started implementing that technology. Of course, it also means you have to have cameras everywhere in your house,

right right, Yeah, they're They're definitely tradeoffs here. It's, like I said, the big one being privacy. Um. There's also been discussions of using the story of technology for things like a t M. A t M so I'm not gonna say that, I was gonna say a t M machines. I apologize for the redundancy. It's funny how that gets into vernacular. But anyway, a t M S could use this to identify a person and then theoretically you could get money out of your checking Accounter Stavings account without

having to have a pen number or anything. Did it again? I did it again twice? An all pen number. Okay, without having to have a pen or any other identification identification on you, you could just it would see your face and know that that was you and identify you with the account. Now, there are some definite problems there because if you have my technical twins, Hey, I'm gonna go get my brother's cash out of his account today. Uh. Also, I'm sorry, No, I was just gonna say, but it

could cut down on skimming. It's it's a hard it's a hard thing to say because I'm reminded of you may have heard the story. It was I think a couple of years ago. Actually in Japan. Japan is always ahead of us on this sort of stuff. Japan had cigarette machines. Cigarette machines had facial recognition technology built into

them to recognize how old someone is. There was this The technology was designed to look for things like wrinkles and and laugh lines and that sort of stuff to identify a person as being old enough to purchase cigarettes, because of course it's a vending machine, so otherwise you're just kind of working on the honor system. But the news broke that kids could easily bypass this just by holding up a picture of an old person's face to

the camera. They just hold up the picture and the camera would detect the old person's face and say, yeah, this person could totally buy cigarettes, and then the kids

could buy as much as they wanted to. Um. So there is a concern about, well, if you had a high enough resolution picture of someone's face and you held it up to the camera, with the camera just be unable to distinguish the fact that that's a a two dimensional representation of a person's face versus an actual three dimensional face, and some technology can't do that very well. But it's getting better all the time, and uh, you know, quite possible by the time this podcast goes live that

will be solved. I would surprise me. I would imagine the best way of doing that would be used to be to use a two camera system where you have essentially binocular vision using a camera with two lenses, and that way you create that whole parallax issue that we have natively as human beings. As long as we have two eyes that work, we have that parallax issue. That's what allows us to to see three D and three

D movies. That's part of the reason. And uh, if you created that parallax, and we're able to create an algorithm that compared the two images so that it could detect whether something was flat or an actual three dimensional object. That would get around that problem. Uh. I just said something that sounds like it's simple. It's actually incredibly complex. But that would be how that would be my approach.

Just you know, you look at how do we see these things and interpret them and then how could we copy that within the realm of technology? Interesting? Interesting? Yeah, good times man. It's um yeah, it's it's one of those things like so many of the other discussions we've had where there are so many positive applications but you know, their privacy concerns and and uh, possible ways to misuse

the technology. It's you know, but I'm it's really fascinating though, how they've been able to develop the technology to recognize is a face at all, you know, especially when it's somebody far away in a photo you have a group picture. Of course, I think my camera only has the ability to recognize as many as six faces at once. Yeah. Um, so there's still limitations, right, some can can do like ten or so, but yeah, it's it's still one of those things that the processors to work pretty hard to

keep scanning and identifying like that. And plus there's a point where you know, if you have a far enough distance from the camera, it's probably gonna ignore it because it doesn't want to try and focus on something in the background at the expense of whatever is in the foreground. And um, yeah, it is really cool. It's it's the it's it's a step toward artificial intelligence. Is really what we're talking about, teaching computers how to to uh observe

things and identify them. They're still not thinking, but they are able to identify stuff. And again, it's really just creating measurements and then assigning a numeric value to the election of measurements and using that as the identifier. So you might think, hey, that's my brother Bill, and your camera's thinking, hey that's six seven four nine dash three to a B or something like that, because it's the identifier that coincides with this list potentially very long list

of measurements of that person's face. Yeah. Now, I wonder who's going to go back through history to identify people, you know, enter it into a database. People like William Shakespeare and Thomas Edison and uh, Nicola Tesla and all these other people, so that it just auto tags everything on the internet. I'm just wondering how many how many cameras are going to go out there and mistakenly identify people's fathers as Kenny Rogers. You know that website, right,

My dad looks like Kenny Rogers. Yeah, so I can't believe you brought that into this. Okay, just say it's apparently a common face to have when facial recognition technology is is everywhere, we will also believe that Kenney Rogers is omnipresent. You know, you gotta no one to walk away, and I think now is the time to run. Yes, yes, don't be a gambler. Alright, guys, that wraps up this discussion about facial recognition technology. I hope that you enjoyed it.

We do have a great article on the side if you want to read more, that goes into more detail about the different the different measurements that these cameras have to take and the methodologies they use. So if you really want to dive into it, I recommend that they also have some really helpful illustrations in there, And if you want to send us any questions or comments, or you just wanna be our buddies. You can check us

out on Facebook and Twitter. Our handle at both is text Stuff h s W. Or you can send us an email or email addresses text stuff at how stuff works dot com. Chris and I will talk to you

again really soon. If you're a tech stuff and be sure to check us out on Twitter tech stuff hs WSR handle, and you can also find us on Facebook at Facebook dot com slash tech stuff h s W. For more on this and thousands of other topics, visit how stuff works dot com and be sure to check out the new tech stuff blog now on the house stuff Works homepage, brought to you by the reinvented two thousand twelve camera. It's ready, are you