Deep Dreaming with Google

seen images produced by Google Deep Dream. But I already know the answer because we spent part of this afternoon looking at pictures of Jonathan in his Renaissance festival costume with caterpillars growing out of his arm sockets and so many dogs, yeah, like bison puppies in his head. It was kind of like It's kind of like the American Kennel Club mixed with lovecrafty and horror, thrown on top of a Renaissance festival thrown on top of a giant

pile of acid. So now you you listeners at home or wherever you are, whether you're at home or standing in line somewhere, or standing on top of a giant statue of a naked Greek hero, someone is I guarantee it probably waiting to hear the end of the Senates. You are now in one of two camps. You're either going, oh man, Google deep dream, this is crazy, or you're saying, what are they talking about? If you're in the latter camp, the people saying what are they talking about? Pause this

right now, Stop what you're doing. Go look up these images. And there are numerous images for you to look at, and uh, and ultimately what you need to know is that the images are all pictures that have been altered by essentially artificial intelligence. Yeah, and in case you're like driving or otherwise visually indisposed, allow me to paint a brief word share for you. You put in a normal old photograph and and what comes out of these algorithms

is recognizably your subject and your background. But the subject might have extra faces in places where faces usually are not, and and the background is maybe dripping with tentacles and eyes, and the edges of things are feathered out, as though technicolor anonomy hats have suddenly become all the rage for animals, vegetables and minerals. It looks you, guys like like like

a gritty post Frank Miller reboot of Yellow Submarine. I was going to say it looks like everything is taking place in the sacred halls of Lord Dagon, and just like dripping tentacled creatures, I go with Lovecraft plus impressionist painters. Yeah, all of these descriptions are valid descriptions. Yeah. I Actually the other night I found someone who had created an app that makes use of the of Google's deep dream algorithm and allows you to submit your own photos and

get them deep dreamed. So I sent a picture of my dog, Charles Darwin, a little Charlie sitting on the couch, and he's so cute. But in the deep dream photo, what's going on with him? Well, the turquoise pillow that he was laying on in the original picture has turned into a giant caterpillar with lots of strange eyes. I see more like a parrot, like a tentacle parrot. Okay, it's kind of a tentacle parrot. Yeah. Charlie's face has turned into a sort of bifurcated evil sweet dog to

face face. And then he's also got a face in his butt which which appears to be a very similar face, almost exactly. There's another weird dog face in his butt. His leg has tentacles and antennae, and like it looks like a fish baby on one foot and then and then a bird where his tail would be. His tail is a bird peeking over the top of the couch. I was trying to figure out what this was. In the original image. It's just some details against the wall. It's like the top of my co feemaker and stuff.

But that has turned into a creepy teddy bear head peeking over the back of the couch cushion angry ewalk. I think it's I think you're essentially making the box art for the next five nights at Freddie's video game. Yeah, so what is going on with these images that you may have seen going around the internet with all these animal faces or they're not all animal faces. That's my favorite version, but we'll explain how you get these different

filters coming through. But there are also some that have just strange accents on curves and corners, or that have geometrical patterns emerging from the figures in the in the picture. Yeah, so what is going on? I mean, has Google just decided to make a super trippy weird art project? Is that the purpose of deep Dream? No, Deep Dream actually is an extension of research that's been going on at Google about image processing that I think is mainly based

in the idea of image recognition. Uh. And this is done through something we've talked about on the podcast before, but we'll go into more detail about today, which is artificial neural networks. UH. And in this case, the the application you could see applications for this beyond you know, making trippy images for for practical purposes, doing things like let's say you've got a picture that has some blurry

elements to it and you've already taken the picture. You can't unless you're using uh, you know, like a light field capture camera. You can't change the focus after you've taken the picture. But you might be able to use algorithms to to recognize elements within a photo and bring it into focus after the picture has already been taken, assuming the algorithms are good enough to do that reliably

and not turn it into a nightmarish experience. Yeah. That is one of the weird outcomes of this type of artificial neural network and image processing is that it could actually lead to the idea of zoom and enhanced I mean, it wouldn't be perfect, but it might be better than anything we've ever had in this fake idea of zoom and enhanced today. Yeah. Yeah, so these beautiful trippy pictures are kind of a mid step between what we have today, which is not zoom and enhanced and and and really

amazing artificial intelligence. Yeah. So let's get into the mechanisms behind what's going on to produce these crazy trippy pictures. And the main thing to talk about is what is going on with artificial neural networks. And I have to admit I have had a lot of trouble like actually

visualizing and understanding artificial neural networks. And I've read about them plenty of times before, but they're they're one of those abstract concepts where it's it's tough to fit it to a real world example that makes it make sense to people who have a i don't know, more intuitive kind of kinetic grasp on things. After after a while reading about them, my and kind of goes, yeah, I'm

gonna go get some sushi, and like that's it. It's it's tricky, largely because there is such a difference between the way our brains work and the way computer processors work. Right, So Artificial neural networks are problem solving systems that are designed to work like our brains. Actually, they're trying to

take computer hardware. Well, actually you could create an artificial neural network that was hardware based, but I think we're talking usually about using software within a traditional computer architecture to mimic the cells inside a biological brain. So if they solve problems by directing data through these layers of

nodes that form information exchanging connections. So let me walk you through, and I'll explain how computer processors at a high level work, and then the difference between that and an organic brain, and then how this artificial neural network is attempting to simulate what's going on with a brain. So, your typical computer process or has transistors, right, They have transistors,

all of them, and transistors are serially linked. So typically you would find a transistor that's linked at most to two other transistors, and these are forming logic gates collectively which direct the ones and zeros based upon very simple rules, and then collectively, when you get lots of them together, you can do neat complex stuff. But they're still linking just to one or two other transistors. Brains however, have neurons along with a lot of other types of cells.

But neurons are interconnected with each other in super complex ways. They're not serially linked, they're linked in parallel, so a single neuron could have connections to as many as ten thousand other neurons. And also, while you look at the number of transistors that are on a microprocessor, we keep on increasing that number by decreasing the size of those discrete elements. So you're talking around two billion or so on a microprocessor, which that's a lot, but our brains

have somewhere around eighty two hundred billion neurons. So we have way more neurons in our brains with much more sophisticated interconnectivity than you would find in a microprocessor. So it's no big surprise that our brains work in a very different way. Now, one of the cool things about our brains is that we can innovate, we can be creative, we can learn things. It takes time for us to learn stuff, but once we learn things, we can then

extrapolate from what we've learned and create new things. And this is where we get everything from. Hey, maybe this would work better if we try it. This way to a genius like Mozart. I mean that's sure. Yeah, this is the basis of imagination and engineering and invention and everything that we kind of when it really comes down to it, talk about as being human, Like what it is to be human is these qualities and there you know other elements as well that may play a part

in this. But our understanding of the brain is still so limited we cannot say definitively like how much of this is what is required for consciousness for example, But that that we've we talked about that in previous episodes, so I'm not gonna I'm not gonna go over that again.

So artificial neural networks attempt to capture some of that complexity and sophistication found in the brain, usually through a software virtualization as opposed to let's hook up these finding eighty billion computers just laying around and trying to connect them together probably not your best use of time. So you're usually going to be creating this through software. Uh. And they have these units, they call them units that

are interconnected um. And you want to try and use these simulations to teach a computer something, for example, pattern recognition or the one that we've talked about before what a cat is, even if you don't tell it that

this is a cat. If you feed enough pictures of cats uh to an artificial neural network, and you use a feedback system so that it is able to different differentiate between cat and things that are not a cat, it then understands that a cat is a thing, even if it's seeing different pictures of different catsum it starts to pick out the common elements to all of these all of these data points that are being fed through it.

Right now, the important part is the training process, because without that training process and feedback, it never learns, right you would, it's meaningless to the artificial neural network. So in in this artificial neural network, each artificial neuron is a unit. There are three types their input units. This is what accepts the incoming information, so that kiddy cat picture, for example. Then you have on the other side of

this network you've at the output units. That's what ends up being the information that says, yes, that is a picture of a kittie cat, or no, that is most certainly not a kittie cat. In between the the input and output you have the hidden units. These are the layers of neurons that represent the various parts of the brain that the inter connections that would happen. Um. And essentially all these units are connected to all the other units. Uh,

And those connections are weighted. By weighted, I mean they have a specific relationship from one unit to the next unit. And it helps to visualize this as thinking of it being from left to right, with left most being input units, right most being output units, and everything in between being the hidden units. So the connections between each unit as you move from left or right are weighted. If it's a positive weight it means that the unit on the

left can excite the unit on the right. All right, So input coming into the unit on the left, it excites the connection to the next unit on the right. Is is weighted positively, it excites the unit on the right. If it's negative, it means it suppresses the next unit that it's connected to that. And keep in mind that each of these hidden units is connected to lots of other units. It's not it's not serial, so it's not just you know, a straight line left right, it's an

interconnected network of these connections. UM, i'm usn't connect a lot. Sorry about that. But anyway, the bigger the way to number, the greater the influence one unit will have on the next one. And a single unit might have all these multiple connections. Some of them are weighted positively, some of them are weighted negatively. The whole point of it is this represents a single sort of think of it almost

like a decision or a perception. So in the case of the kenycat picture, the first wave might be very general shapes that would be associated with cats, and then the next wave might be more particular details, and the next wave more particular details. And uh as units pick up on those details and send the message on further down the line, it starts to refine it and refine it until it finally comes to the decision of yes, kitty cat or no, not a kitty cat. I'm way

over simple but but yeah yeah. So so so there's there's a bunch of layers in the middle here where the machine is going like, yeah, this is probably a kitty cat. Yeah, yeah, probably or probably not until you get to the very end and and generally you have like a threshold and if the data at the end of it meets that threshold or exceeds it. Then it's one result, and if it doesn't, it's a different result. It's a negative result.

So you can almost think of it as the probabilistic approach that a system like Watson goes through when it's trying to determine if an answer to a jeopardy question, or rather the question to a jeopardy answer is the appropriate one, where it says, all right, as long as it meets this level of of of being sure this is the correct one, we're going with it. It'll push the button and yeah, give give an answer to the form of question. And maybe when a copy of the

home game. Uh so, all of these, all of all of this is going in what is called a feed forward network, which is just one type of artificial neural network. I'm using the feed forward network because it's one of the easiest ones to explain. Uh there are others that get way more complicated than this, and it requires an understanding of artificial neural networks that goes beyond my surface

shallow level of understanding. Now, one of the ways that artificial neural networks have become most significant is in the field of machine learning, where you're not just coming up with a logical process for a machine. But you're showing a machine how it can refine its own decision making. And that comes in with the feedback that I was talking about earlier. You have to have feedback. You have to tell the machine. You have to be able to communicate to the machine when it has made a success

versus when it has failed. And you have to be able to tweak the machine. And by machine, i'm talking about software in this case, you have to you have to tweak that design so that you get the outcome you want. Now, this is where we get a little meta. We as people know if a picture is of a kittie cat or not when we look at it and we recognize it whether or not it's kittie cat, well as an adult human who has experienced cats. Yes, yes, don't over generalized, Jonathan. Sorry, my my one year old

niece knows what a kitty cat horse. All right, Let's let's say that we have determined ourselves that this this photograph we hold and on multile hands is that of a kitty catch And I'm sorry to want you know what we're gonna We're gonna title this episode after a Hamlet quote. It's going to be called very like a whale, uh, because it makes sense in that context. So anyway, you've got a picture of a kid cat. Now you you feed it to the computer and the computer output comes

out and it says it's not a kiddy cat. And you know that's the wrong answer. So you have to look at how to fix the system so that it recognizes the picture you're showing it is in fact a kitty cat, right, And that might require you to to dig down back through those layers and and pick out the one that kind of said like, noah, well it's

more triangular, so it's obviously not a cat, right or whatever. Exactly, you have to figure out where in this, in this stage of interconnections, did that one decision or maybe multiple decisions lead to the conclusion that it was not a kitty cat. The one one way of doing this is called back propagation, where you start with the output and you work your way backwards and you say, all right, for in order for this to say yes, this is a kitty cat, we need to have this result at

this stage. Do we have that result? Yes? All right, let's go one step you know actually probably no, no, Well, then what is going on the step before it, that sort of thing, and you work your way back and you start tweaking those waitings. I was talking about the connect connections, and you say, all right, well, maybe this connection is actually waited too much. It's too far in the positive. We need to bring that down. Or maybe it's in the negative and we need to switch it

to positive. So you start making these adjustments in the software to those weighted connections in the neural network, and that might end up allowing you to pass that same kitty cat picture through and now it says, oh, that's

a kitty cat. Like Yeah. You do this a lot, with lots of different examples, and eventually you get to a point where you feel confident that it is doing what you intended it to do, that it is in fact able to recognize the picture of the kitty cat at a high enough percentage that you're that it's you

that you say, this can recognize a kitty cat. Then you can start feeding it pictures you have never shown it before, including pictures of stuff that looks like a kitty cat but isn't, and kitty cats that maybe slightly outside the norm of what it had experienced before, and see how it does in that case, and once you get to a certain point, the device is able to maintain its ability to recognize things without you having to go in there and tweak stuff in between the training

sessions it has been trained. Yeah. So so that's what they were working on, was the idea of image recognition. But one of the things that comes out in the Google research blog post where they were first describing the the idea and the genesis of the Google Deep Dream was that the researchers found that and I'm gonna have to quote this here, neural networks that were trained to discriminate between different kinds of images have quite a bit

of the information needed to generate images too. So that in training, and that was the end of the quote, So that in training, what they had done in teaching these neural networks how to recognize images was also sort of teach them how to make images of things. Right. If you say, here are the shapes that you would see in Japanese architecture, and these are the these are the shapes commonly used in that architecture. So this is how you can recognize you're shown an image of a

building from historic region of Japan. Then it knows it being you know, knowing and is being generous here, but it recognizes those features. Those are the features that define what that thing is. It can now generate those same features. And so if it sees quote unquote sees patterns within an image that resemble that, it could generate those images, kind of kind of tweaking and shaping the the fed

image and producing something new. Yeah. So they had some wonderful examples on this Google research blog post where one of the things they were doing was just refining images based on white noise until they started to show the

image that was desired. So you would start off with static, just pure static in an edge, and then tell the algorithm to constantly tweak that static to enhance it to become more like an image of a banana, and eventually, for example, yeah, yeah, and eventually the static evolved into a banana or a cluster of bananas kind of I wouldn't say, like like not like a group of banana, but a banana pile, like some sort of weird minion

slash banana box made out of bananas. And I thought this was funny because this is the digital equivalent of apophenia. Do you know the process of apophenia. It's in psychology where we see significance in random patterns, sort of like paradolia being a very specific version of that, where you can you see faces in in shapes, like seeing something

in a cloud, very like a whale, very much. But yeah, so we were essentially here teaching computers how too, Well, just keep trying at all of this random noise until you can find the banana there. Yeah, it's almost like pointing a sculpture at a block of marble and say, just keep cutting away until the masterpiece, like David emerges, until you find the banana. I'm sure that is what someone told Michael Angelo is a very young boy. Pretty sure that was. That was in one of his famous paintings.

It's just there's a little thing at the very end, like have fun, find a banana. But yeah, obviously the actual goal of the research is image recognition. I mean we've actually done podcasts about image recognition of various forms before. Oh yeah, well that that in speech recognition and and facial recognition, which is very uh kind of creepy and important in our daily Internet lives. And if you if you want to, you know, we've talked about this so often.

If you want to do a really deep dive on these topics, you can check out episodes such as can computers describe what they see? From November? I know that face from October zoom and enhanced from August, and speech recognition from April. Man, we talk about computers learning a lot. It's well, it is the future and and there's still

a lot of challenges to this field, right. I mean, it's not so easy to make a computer see and recognize what it sees the way a human or an animal would well, and a lot of these involve uh systems that are tweaked for very specific types of recognition. It's not like you have one neural network that recognizes everything. If you looks really just the cat network or the banana network. Yeah, by the way, I get the banana work network. It's really appealing. So how are you guys

just released acknowledge that entirely? I think we keep that whole part. I think we do. I think we should continue right now. Okay, but but so if you if you want to go on the teap dive, So if you want to go on a deep dive about all of this stuff, you certainly can. But let's go over like like a basic overview of why it's so difficult to get computers to to see and hear the way

that we do well. The big one being that architecture, you know, the difference between computer architecture and the way our brains work. That's that's the biggest, right, that's just fundamentally they work in very different ways. And we have expressed this in multiple episodes too, especially dealing with things like how computer memory is so different from our memory.

That's just one easy way of pointing at this. So that's a big one even And the software simulations are incredibly limited because they require a great deal of processing power to work properly. Um, and uh, you know, we're still learning how the brain works, and so to create simulation of it while we have only a partial understanding

is really tough. In fact, a lot of people, I saw one person say, uh, the simulation the artificial neural network is similar in a way, like you wouldn't say it's a brain, the same way you wouldn't say a weather simulation is an actual weather front. It's you know, it's a It's as close as we can get right now, based upon our understanding and our technological sophistication, and that's going to only improve as time goes on. But we're

still at the very early stages of that, sure. And it's also just a major difference in the approach of problem solving. I mean, typically, computers as they exist today are good at learning by explicit instructions to get the right answer. Yes, and assuming that everything in the computer is working properly, then they will reliably execute those instructions precisely every single time. Yeah, And our brains are exactly the opposite. They're not getting the perfect answer, but they're

very good at something computers aren't at approximating. They're good at approximating based on a lot of inputs. So we learn what a chair is not by reading a definition of the key features of a chair, and then a list of all the possible exceptions, including every variation on a chair there could be, I mean, like, why would we do that? Instead we just see a bunch of chairs. Notice that people identify all of these things as chairs. We can generally yeah, and then we get an approximate

idea of Okay, here's basically what a chair is. So we have a sort of fuzzy feel for what constitutes that object. And that's what the neural networks are trying to do. They they have large samples of data and they try to get a feel for it. And it it takes the the the work of actual human beings to make certain that that early stage of training is

actually working. It's not like, it's not like we have a computer that you can just turn on and it just automatically starts to learn and it knows when it's right and knows when it's wrong, and it can thus start to learn everything. We're nowhere close to that. We are to the point where you turn a computer on, you feed it some information, you see what comes out, and then you either say, all right, looks like this particular uh go through work. Fine, let's try something else,

or you say, oh, this didn't work. Let's find out what's wrong and fix it so we can try it again before you ever get to a point where you can start showing it new stuff, right and kind of that. The way that you do that, that you build a better neural network is that you you check in on what it's doing, and by by asking a layer to create a visualization, a layer of these these artificial neurals to create a visualization of what it's working through. Is

one way that you have of checking in interesting. As an example from this Google research blog, you might have set your network to figure out what a dumbbell is, but until you ask it for an image, you might not realize that all the pictures of dumbbells that it's found so far include beefy human arms. So it's obviously searching through our stock image libraries that we use. Sure, sure,

but you know so. So once you see it's images of these really weird arm dumbbell hybrids, you can help it correct by telling it to enhance the bits that are dumbbells, you know, the shapes and the colors that

go with dumbells, and to ignore the beef arm bits. Yeah, it actually is very interesting because as much as I joke about the stock stock image thing, it really does tell you that when we choose certain images to represent concepts, we often will go to very similar ones, and to the point where as humans, we know we can differentiate the thing in that image that actually represents the concept versus some other supplemental thing. But a computer doesn't know

that unless Yeah, and it's so loaded with context. I mean, you wouldn't know why a bf arm and a dumbbell would go together, and you understood culturally speaking that that sometimes when you work with a lot of dumbbells, you get big BPRM. So all of this is extranees information that a computer can't possibly be asked to automatically know

the way that a human person would. Uh And okay, So so extrapolating out a little bit further from this concept, the end goal is really for your neural network to be able to auto correct, so you can program it to enhance whatever it thinks is important, and it will look for patterns in the visual data of an image and enhance those, then evaluate the resulting image and find more patterns and enhance them, and so on. It's a little like like asking a child to tell you what

shapes she sees in the clouds. You can get a decent sense based on her answers of her abilities to think abstract lee and to extrapolate visually interesting. So yeah, yeah, because I think of ties where uh, me and my friends would be looking at the clouds and we would sit there and talk about what shapes we saw. And I remember like a friend might say, oh that I see a dog, and other guy says, I see a man, and they'd say, what do you see? As I see it looks like it's going to rain, and m hm

explains why I don't have friends. But but yeah, you know, so so you get to you get to evaluate this kid's conceptualizations, and you also probably get to have a really rad, trippy conversation. And these are the two things that we are getting out of deep dream. Yeah, yeah, certainly. And to bring it back to this sort of byproduct these deep dream images, how you actually get these is

that refining feedback process. So at a certain layer of analysis, you tell the neural network, okay, what whatever you found here, focus in on that and and pay a lot of attention to it, and then look at it again and then pay more attention. Yeah, and and it does. Paying

attention goes beyond just focusing. It goes to the addition of information, right if relation of information Yeah, yeah, Like if you think that you see a pagoda in those clouds, then really really enhance that pagoda tendency, right, And this is how you end up with Jonathan having eyes all

over his shoulders and caterpillars for arms. It might be looking for images that it has recognized before in biological life, in pictures of animals, pictures of insects, and say, yeah, okay, that arms kind of you know, kind of tube shaped. So maybe we can make that a little bit more like a caterpillar. Oh now it's looking a lot like a caterpillar. Make it look more like a caterpillar. Even, Yeah, totally.

By the way, you can get to that same destination just by hanging out with me on a Saturday night, Same same result to me. This actually raises a pretty weird and possibly interesting, possibly superficial question about artificial intelligence. If we're teaching our computers to seem or like us, does that mean they'll eventually learn to hallucinate like us? Like Is hallucinating a natural consequence of human levels of vision and object recognition? Well, I would ask, how how

do we know that they're not already hallucinating? There's a theory Follow me here, Follow me here. There's there's a theory of baby brain growth that suggests that infant sensory processing is on a level similar to adults that are using hallucinogens, because there's there's less or or even no awareen awareness of context, and there's less separation between internal

and external stimuli. So everything feels and looks and sounds real and immediate, including things that are artifacts of brains inner processings like associations and fragments of memories and misunderstandings of what you're seeing and hearing. So h so are our computers all tripping all the time? Is the question I posed? Do androids dream of electric sheep or are the electric sheep there for them always? I don't know. Well, they don't dream of electric sheep standing in a pasture.

They dream of electric cheap emerging out of your pectoral muscles again Saturday night. Yeah, this is ah to me? This is this isn't a great A great way of appreciating how weird and amazing artificial intelligence as a discipline is. And while while this is almost like a byproduct, like its just an interesting byproduct of something that was intended to improve upon image recognition software, it also has created

some truly remarkable images. I mean, it's it's you could argue it's a new form of art, and you start by feeding it an image that you think is already interesting or maybe not interesting. Because either way it works and in you see what comes out of it, and U we've even seen some pretty you know, mostly it's done for laughs, but there have been some that I think it just really striking images that make me think of Impressionism and some other and more even more abstract

and surreal approaches to art as well. Some of the ones I've seen very much have a Salvador Dolly kind of the ones with like a dripping architecture where there seemed to be you see, like arches and things that you would recognize from buildings, except they seem to be made of liquids somehow. Yeah. Yeah. And there's also an element like Dolly of that, of that kind of of that kind of Escher sort of influence in there too, because of the way that the shapes repeat and twist

on each other. And oh it's fascinating, like impossible perspective. Yeah yeah, yeah. So we're getting some some phenomenal pictures, uh, and maybe we'll even post some from the Forward Thinking crew later on. We've talked about the possibility of of doing a photo shoot just just to feed it through here and find out what kind of fresh horror awaits us.

But I really think that this was cool also just to kind of get a look at how artificial neural networks work and and the process that they tend to use in order for machines to be able to learn stuff, because we've talked about learning so much without really getting into the process that's going on, just you know, to say that, hey, this machine can learn. It's it's it's not it's doing it a disservice. So this was great to get into the nuts and bolts of that. I'm

going to go home and lay on a caterparrot. Okay, well you you own one. We've seen it. So guys, if you have any suggestions for future episodes of forward Thinking, maybe there's something you've always wanted to know more about, like what's that going to be like in the future, Or maybe there's even a topic we've covered in the past that you want to have us focus on the very specific part of that, or whatever it may be.

You should write us, and maybe you just have comments about the the goal project or artificial intelligence of machine learning. Send us a message. Our email is FW thinking at how Stuff Works dot com, or drop us a line on Facebook, Google Plus or Twitter. A Google Plus and Twitter, we are FW Thinking or Facebook. Just search f W Thinking in the search bar. We will pop right up. We can leave us a message and we will talk

to you. Against will be soon. For more on this topic in the future of technology, I visit forward thinking dot Com, brought to you by Toyota. Let's Go Places,