VAEs Are Energy-Based Models? [Dr. Jeff Beck]

Build the symmetries in. Unfortunately we've got a lot of great tools that were developed over the last several years that can do

Um it you know uh it has uh sophisticated policies that are context dependent, which is basically saying really long time scale. Um and things like that.

Right. So for example, when we think of how the difference between like us and like like really like amoebas, we we often cite things like planning. Counterfactual reasoning, goal-oriented behavior, right? We're specifying things that that um have that that are specific mean that that are all related to how it is we compute our policy. Right. They're latent variables that represent policies. Um that are uh, you know, that are compatible with like well reinforcement learning. Right. And um

And that's the defining characteristic of an agent. But you could very easily just sort of say like from an outside perspective, if you can't look at how someone or something is doing the computations, if the only thing you observe is the policy. Right, does that mean that you can never conclude that something's an agent? And I would say no, right? You'd still like to be able to conclude that this is an agent, even though the only thing I ever get to measure is its policy.

Now, is it normative? No, it's not. It's it's a but it is a measure and you could use things like that. But at that point you're really just talking again about policy sophistication. Right. Not does it have a reward function? Like is it actually executing planning?

Right. So all of those variables, all of those variables that were that occurred inside, right, from the outside perspective, it just looked like a function transformation. Right. It it's I don't unless I unless I'm somehow going in and recording and somehow demonstrating the fact that the manner in which it is calculating its policy re you know, like involved doing those rollouts.

Right. I wouldn't be able to show that it's actually doing those rollouts. I would just be able to conclude it has a really sophisticated policy. So can you conclude that something isn't is is so so the question is how do you identify something as actually doing planning? And I think that's a really hard question, as opposed to having an incredibly sophisticated policy.

Right. Because you know it's going on, you could say, oh well it's it's clearly like executing, you know, this is it's doing planning and counterfactual reasoning. It's going on. Like look, there it is. Because you coded it, so you know it's doing it. But if you're looking at it from the outside. Right. It r uh you know, if you don't know what's happening inside, it's going you know, all you have access to is oh well here's the action that it w that it c that it did given this long series of

And so it's it's really hard to identify what you know, something as an agent per se from the outside. You kinda have to know what's going on in This by the way is why I don't think that like, you know, can you know, these sort of prediction based approaches to like AI

um are necess you know, that you could sort of say, well, it it's not really doing anything even remotely agentic unless it's executing it's doing planning and counterfactual reasons. So like your chess program is is like, oh clearly it's doing some planning and counterfactual reasons. because you know it's doing it. But um but it but you could like write yeah I could describe the exact same set of behaviors just with the policy function.

then you can draw an implicit conclusion that, oh yes, well, I may as well say it's an age. And that's kind of the approach that I've taken. So like one of the things that comes out of the physics discovery algorithm is that you apply it to agents and what do you get? Well you get a model. Now bear in mind I called them all objects before and I didn't change anything to make it special to an actual agent.

Right. But what I do have the ability to do because of the model is I can look at the internal state. associated with that object that I want to call an agent. And look at how sophisticated it is. Right. And that degree of sophistication is what allows me to say, oh well, I'm going to go ahead and say that like and I like the whole idea. It's a great idea. Like it's have a metric.

Right. And I'm sure it would be something that would effectively be like transfer entropy or something like that. But like we have this metric on like, well, how sophisticated were the internal states that were necessary in order to generate this output? And if it's above some threshold, we'll call it an agent. I don't like threshold. But, you know, we just sort of say a degree of agency, a degree of sophistication.

Right. This is this is the best model. Right. It's not the it's not th this is why like I I I repeat this over and over again. Grind it into the students, right? Is that that, you know, science is about like prediction and data compression. and like nothing else. And the same thing is going on here, right? You you'll never n you know, just looking at behavior, you'll never know e g for sure in any meaningful way

like whether or not it's it's just doing a function transformation or whether it's engaged in planning and counterfactual reasoning. But if your best model of it

If you sort of say, well, I tried to model as a function transformation, but goddamn it, it had a lot of parameters. Right. But then I tried to model it as something that was just doing Monte Carlo tree search on the inside and giving the answer, and that had like, you know, 40 parameters. And it's like, well, that's the model I'm going to go with, and now I'm going to call it an eight.

Right. Philosophy is really about drawing clear lines and distinctions. And in my world those don't really exist, right? There's everything has an error bar. You know, all of i there isn't a clear delineation between you know, uh you know, an object and an agent. It's really you know, in from this modeling perspective, it's really just a question of degrees and philosophy is terrible at handling questions of degrees.

Right. It it basically, you know, energy-based models, you know, uh if you're just doing function approximation, you're basically saying there's any mapping from X to Y, X is by inputs, Y is by any mapping is out there, I just want to figure out what it is. Right. Now in an i you know, in an energy based model, right, you're you're you're you're effectively placing.

on what that input-output relationship can be. I like thinking about the distinction between an energy-based model and a and a traditional sort of feedforward neural network has to do with where your cost function is applied. Right. So in a in a traditional neural network, you take in your inputs, you got your outputs, and the cost function is just a function of the inputs and the outputs. And the only thing that you're optimizing is the weight.

In an energy-based model, there's another thing that that your cost function operates on, and that's something, one of the internal states of your model. And as a result, like in order to figure out what the best, you know, the best approach is, right, you actually have to do two minimizations.

One that that finds the energetic minimum associated with the the the part of the the cost function that operates on the internal states, like the hidden nodes of your network. Right. And then one that is the prediction that is your like effective prediction error.

Um this is this is very much consistent with the approach that a Bayesian would take, right? You have a you have a a a prior probability distribution, which gives you an energy function over every single latent variable in your model, and you are optimizing it with respect to all latent.

So you take a probabilistic progeny. Good examples of this are like a variational autoencoder. A variational autoencoder, I think, is the gr is is the best example of the most commonly used energy-based model out there. Why? Because you have an encoder network, you have a decoder network.

Right. And your cost function is based on the difference between inputs and outputs, right? So that's just like a that's fine, that's still a regular no but it also is how how Gaussian in it well it depends on what flavor of VA. But you also have some c uh some some part of your cost function um is a function of the actual repr internal representation. Right in a traditional VAE it's it's how Gaussian is it? You want that internal representation to be as Gaussian as possible.

If it's a VQVAE, then it's like mixture of Gaussians. But it's still like a cost function that is applied on the internal states as well as on the inputs and outputs.

Now, I think that like which which makes it a great trick, in my opinion. It's like, oh great, like yeah, they're they're they're they're they're moving in the direction of energy-based models. I love it. The only thing I don't like about test time training is the vast majority of the training that is done. So in a traditional energy-based model,

You always find the minimum with respect to the latent variables, right? These extra weights that you know, which in this case, which in the case of test time training is the you know, the subset of weights that you're allowed to to change during you know during test time. When you do the training for a traditional energy-based model, you're allowed to make those changes right throughout the entire course of training.

The way that we're often doing test time training these days is we just do regular old neural network learning, like we don't do and and then and then and then finally when it comes to when we get to the deployment phase, then we suddenly turn on Right. Th these additional latents which are basically some of the weights of the network.

And we do additional an additional bit of learning at that point. This seems monumental now, again, not an expert here, right? But this seems unwise to me. And the reason it seems unwise is because you didn't train the original network with that on. Right. You trained it as in a completely supervised way. Yes. Now I'm sure that people have s are aware of this and have it's been addressed in the literature, but I'm not personally aware of that. I don't think that's how it's used in practice.

And there's so much information in the test sample. Let's just riff on the relationship between energy-based models and and Bayesian inference. So of course they have this advantage that you don't need to do this for expensive and tractable normalization test.

And so the difference between uh you know, like w which is sort of like s you know, in a Bayesian framework, that's like saying, well, you know, I'm not actually gonna treat some of these latent variables i in a probabilistic way. I'm just gonna do maximum or map estimation.

on some of my variables and just be okay with that. And that's one way to interpret the relationship between an energy-based model and a properly Bayesian model. There's there's a happy medium here though, right? And the happy medium is you can still treat it.

As if it's, you know, you know, you don't have to just minimize the energy function, but you can calculate the curvature down there too, do a Laplace approximation, and call yourself a Bayesian again, right? Yes, there is more computation involved, but we've got a lot of great t tricks for making that totally tractable.

Now, if you're just doing maximum likelihood estimation, if you're minimizing your energy function with respect to some particular well just well, let's pretend we're only we're at one variable. Um, and I'm just gonna like get a point estimate and call it a day, do like, you know, some kind of map estimation to get to get that that one thing.

There's not that big of a difference, right? Because you're you're not there is no probability distribution over the latent that allows you to compute that regularization term. But that's the only difference. It's it's are you regularizing or not? Is uh I think the easiest way to think.

Yeah.

And that's a great idea. It's a great idea because it has some of the flavor of what we would like to get out of our models. In many situations, we're not interested in predicting every single pixel on the image. we wanna get, you know, maybe something that's a little more gestalt, a little more high level, a little more conceptual understanding of what's going on. And so emphasizing the goal of predicting every single pixel, which is what's typically done in generative modeling right now.

You know, might lose some of the power, the abstractive power of some of the networks. And so like, let's do so so the whole point of JEPA, as I understand it, I'm sure there are other points. um is that uh is that you're gonna take the you're you're gonna you're gonna compress your inputs and compress your outputs and then do all the learning in this compressed space.

Love it, right? Science is about prediction and data compression. Let's make that compression explicit on the front end and the back end. The downside of this approach is that is it is it it doesn't work out of the box, right? Because it's very easy to find a compression. Or an embedding of the inputs and an embedding of the outputs for which prediction is perfect. Yeah. Which is to basically make both of them zero.

And so you have to do some other things. Other tricks need to be employed in order to make it

And in many different ways what they're trying to do is avoid this mode collapse problem that you're talking about. And they use different forms of regularization

Right. And so what do we do? We run a VA that we do the pre-processing. And we do it by the pre-processing step is completely independent. Right, from the actual algorithm that's gonna be the be be be tasked with solving the problem of interest. Um and you know That's not something that we necessarily have to stick with, right? It would be very nice

if there was a way uh i i if there was a way of like again, well uh jointly. Ha we're getting right back to JEP again. What we'd like to do is we'd like to choose our pre processing algorithm in a manner that that the it you know you know uh not a priori, not do it first. We like to choose the preprocessor that works the best in in this space.

And I think that that's the ultimate motivation for a lot of this work is that it's like what's the right embedding? One of my favorite tricks, like of course I you know, I pre process the VA's all the time. In fact it's when you know, the second w every time someone hands me a new neural data set, the first thing I do and I'm

Right, to to have a single algorithm and approach and not just be stringing these sort of things together in an ad hoc way. There's, you know, when when doing PCA, PCA is a really great example of this. There's a failure mode for principal component analysis.

Um, which is actually really common in neural data because principal component analysis basically says, well, where's the most variability? Okay, I'm gonna worry about that. And then all the stuff that's not varying very much, I'm just gonna throw it away. Right. It's just like look, you know, dimensions in which there's low variability are not important. Well, it turns out that in neural data, the dimensions in which there's very little variability are some of the most important dimensions.

Yes. And so pre-processing with PCA runs a risk of throwing out the most valuable information in your data set. Yes. And so there's a lot of wisdom in in in jointly, right? Pre i in in jointly fitting your pre processing model as well as your inference and prediction.

um the neural network gets wise and what it does is it kind of discards all of the the the l the long tail stuff that aren't relevant for that particular task. So when you train these models, what you're trying to do is sort of maintain enough ambiguity So that it it compresses the information but it also maintains enough fidelity to work broadly for different things.

Um that said, like the vast majority of what you know the brain does, just like these neural networks, is decide what information is currently task irrelevant. But that's all the more reason to do things in a self-supervised or unsupervised way, right? Because you're basically not telling it this is the important stuff, you know, you're not telling it like what's all task relevant.

They're working on version three and there'll be version four, there'll be version five. Will there always just be something left over?

Right. It's sort of like we get algorithms that do amazing new cool things and then someone comes along and says, Yeah, but it can't build me it it can't pull a rabbit out of a hat. Right. And then and then of course what does someone do? They oh they They figure out the a new training protocol, a slightly different architecture, or they just train it to pull rabbits out of hats and then suddenly it can't.

And then someone proposes a new challenge and a new challenge and a new challenge. And it's always this game of like one-upsmanship. So the question becomes, well what's the point at which there are no more new challenges? And I'm not entirely certain we're ever gonna get there, right? Um it may very well be the case that we get uh you know, these sort of

algorithms that are capable of replicating the complete suite of human behaviors and then someone will come up with some criticism like, yeah, but it's not really doing X. It's just faking it, right? This is just the direction things go because people really do think they're important.

Right. So this is like the is you know, and there are some approaches that it's all based on like Bayesian nonparametrics and Diersley processed priors and stuff like that, where you you sort of see something that's surprising or unique or different, something you didn't expect. And it causes you to say, I need to turn learning on because I gotta figure this out.

That is an absolutely critical element that we need to be developing. We are developing that. And it turns out that that's one of the nice things about this sort of object-centered. Physics discovery thing is because it's object centered, if it comes across a new situation that it does not understand, it is capable of instantiating a completely brand new object just to explain this new situation.

of something that at least in principle is a generative model of generative models, right? It's sort of like, oh, like you know, I might actually need a new node. Like it's time to create a new latent variable because like like the current set's just not cutting the mustard anymore. Those are things that that that I think are hallmarks of of true intelligence. I don't want to ever make the statement, as soon as it's got that, it's truly intelligent. I will never, ever, ever say that.

Um but I do think that that is a a a a critical component that that needs to be present, right? Is the ability to generate new models on the fly to deal with novel situations and data. Um most of that, you know, w um you know, as well as the ability to um uh combine old models, previous models in new and interesting ways. This is actually how the brain evolved, right? We started out with like um

you know, really simple brains and there were different regions and they solved sort of different problems. And what eventually happened as we evolved is that these different regions of the brain learned to communicate with each other in new ways. And through that communication acquired new abilities, right? And then eventually evolved into in, you know, you know, um new capabilities and things like that.

Right. I I often like to point out to the the ol uh I think olfaction is like the the sense that's not studied nearly enough. It's an incredibly old part of the brain. Um and arguably, right, it's the it's the first part of the brain that evolved the ability to do proper like associative processing. Right. Odor the odor unlike visual space, right, where there's translation symmetries and and all that sort of stuff and things are smooth. Olfactory space that does not exist, right? It's it's

Really, really, really combinatorial and complicated. And the part of the brain that evolved to solve the olfactory problem arguably is the part that evolved into our frontal cortex. Don't quote me on that. There's a lot of disagreement there. That's just my take.

Um, but it certainly has a lot of the features that we associate with associative cortex, right? It is it wow, I just said g got like six uses six three different uses of the word associate in that sentence. But but y I think you see what I mean, right? It it um It was all about like taking old capabilities, right? Combining combining, you know, simple models and modules to create something that was more complex.

And then over time, right. So so that was what made the brain work, right? It was all about taking little things that work. and combining them in new and different ways in order to evolve, you know, uh effectively an emergent, you know, emergent properties, emergent calc, you know, computational abilities and an emergent understanding of the world in which we live.

And I do think that like what what you know, if when we get to the point where we start really saying, oh, this is actually truly intelligent, it's going to have that feature. It's going to have the ability To have a m- It's gonna have a modular description of the world, and it's gonna have the ability to combine those modules in a way that creates a more sophisticated understanding.

It's like Legos, right? I can, you know, the the Lego bricks all connect in certain ways and I can build like all sorts of new and amazing things that were never built before. right, out of them. That's a capability that we have. And that's the essence of like creativity. It's why I refer to systems engineering as like the thing we really want our our our AI models to be able to do.

and their ability to learn how to to to work together that that uh you know gave rise to the technological revolution. The brain is the same way, right? It's i in my view. It's highly specialized little modules or agents that are capable of of of uh of of um being repurposed, reused, um, capable of communicating with one another in order to solve really complicated problems. But there's always a benefit to specialization.

I don't believe in like like AGI. AGI seems like a bit of a a misnomer to me. What we really want is not artificial general intelligence, we want collect we want collective specialized intelligence.

Um and uh the next step of course is design i i is constructing a system that tests these hypotheses explicitly, right, and generates the experiments that will identify like that will the fill in the gaps of our knowledge. And all of this I believe can in fact be automated in a very sensible way. I d I d you know, I d I don't see any like major obstacles to automating empirical inquiry.

Other than we probably want to place some safety constraints when we start letting them work when the we start letting the AIs run the labs, right? Because you never know. It's sort of you always have this AI where it's like, well, you know, the most effective experiment to determine if this is correct is to set off a nuke. And that that would be

Yes. Right. So pure empirical inquiry, right, does run risks like that. But I think that that's not not not the biggest issue. I think what we need to do is we just had need to have a nice concise framework for saying like, Oh look, you know Like I'll give you an example. So we had this we we we had this um a problem that popped up a while back.

A gentleman we were talking to is is um is you've got these laun y you know, you've got these robots and the robots see something they've never seen before. And in an idea, you know, so a robot is like running around, it comes across like a beach ball, never seen a beach ball in its entire life. And what you'd like is you'd like the robot to know how to figure out that it's a beach ball and to figure out what its properties are.

And if you tell the robot like like if you see something new, just stop. Right, you're kind of then that's that's no good. Right. What you really want to do is you want to figure out a relatively non-invasive procedure for the robot to like poke d do what a child would do.

What does a kid do when they see a beach ball, right? They run up and they poke it and they say, oh, right, yeah, and then it moved and and it it actually learned it actually experiments with its environment for the purposes of identifying the properties of the objects that exist in it.

Um now I do think we probably want to test this out virtually before it's deployed in the real world because you never know. It might very well be that the optimal experim experiment is to run up and kick it as hard as you possibly can. Um, and we we certainly want to avoid that. But like something along those lines. Something you know, a robot that is able to test the theories that it has um about how things work.

in an online way and learn from those results in an online way is definitely part of the goal.

I do want to see a future where where where we have an AI that actually improves our understanding of the world. And simply automating everything runs the risk that you specified, right? It runs the risk of people becoming couch potatoes that just watch T V and occasionally say, like, yeah, you know, these chips are no good. Um uh that seems like a bad outcome to me. Um I worry less about that, I think, than some because people are remarkably adapted.

Right. I mean I you know, you they have all these arguments about like, oh, you know, this new technology comes along and it's gonna completely destroy this way of life and you know, and that's gonna be awful for people and it is maybe in the short term. Um, you know, I think of like tractors, right? Or just go back how many hundred years do you have to go back when like ninety-nine percent of people were involved in agriculture and now it's like what two?

Right. I consider that a solid improvement, right? Because it allowed the rest of us to d it allowed us to do a bunch of other things that we find more satisfying that are more interesting. It allowed us to like, you know, like I can read, you know.

Spend some time reading a book, don't have to labor in the fields all day. Um, that's the future that I sort of see, and that's the future that I hope for. Is that is is one in which You know, all of these artificial agents running around and doing things autonomously are there to free us up.

to pursue more interesting and more, you know, you know, to improve ourselves in in in in in more interesting ways. But at the end of the day, it's just another technology, you know, at least initially it'll just be another technology like the tractor. Um now a hundred years from now, who knows?

cybernetic transhumanism, if I'm gonna go sci-fi on this, right? Where where, you know, the technology and us evolve together in a way that's beneficial for both. That's the goal. Um, you know, are there these dystopic possibilities where like, oh, well, what are humans in a world where well what are they? What are what are humans in a world where everything can be done by a robot? Yeah. You know, that's that's a good question. And th that's and at the end of the day, right.

they end up just becoming like reward function selects. Right. They end up just sort of saying, Oh, I don't like this and I do like that and they're basically, you know, I mean, you end up with a c is another nightmare scenario. I don't like talking about these dystopian futures.'Cause honestly I think people are too clever.

And I think people are too motivated and people are too interested in how the world really works and that people are too interested in actually understanding things, that they will never stop, that they that AI will become a partner, not an adversary or a crush. And that's that's that's what I think will happen because that's but that that's a statement more about my belief about humans than it is about my belief about the development of AI.

You know, I am a techno optimist, if you will, not a not a pessimist. I I believe that we will find a way to adapt to an ever changing world as we have done for millions of years, including one that includes technology that alleviates most of our labor.

Right. Um, in part because, you know, AI is not that advanced, but also because we are telling A we you know, we are still in the position where we specify the goals of the system. And that will likely continue for a very long time. And it will always be the case that these systems, you know, will can be you know w

are are subject to review. We will always keep an eye on them. They will always at least initially be be be released in relatively restricted domains and where we're where we're test where where we're keeping a a close eye on what it is that they are and are not doing. So I don't worry too much about like the going rogue. I worry a lot more about somebody who

building, you know, it's sort of like a virus, which we already have to deal with. Like somebody builds like some insane virus and like takes down the internet. I'm more worried about malicious human actors than I am malicious AI actors because at the end of the day, all of these algorithms, they simply do what they are told. Right, we train them, we tell them here's your objective fun.

As long as we are specifying the objective function and we understand the objective function, we're probably going to be okay. I think the safest way to deal with AI concerns is to tell people, hey, look. This AI is just doing what we told it to. We train we we you know we we set it up to make really good predictions and to achieve these outcomes. Now is it dangerous to like specify these outcomes without being very, very, very careful? Yes it is.

Right. That's this is the whole like hey Skynet endworld hunger and it kills all humans. That that's a that is that that is a real possibility. But whose fault was that? The fault was the person who like was very, very naively specified their goals. There are in fact relatively straightforward ways to specify the reward function that that don't run that risk nearly as badly.

And the best one is so are you familiar with like maximum entropy inverse reinforcement learning? I like to call it active inference because it's really similar. Um and so there what you're doing is you're basically observing someone's policy and then you're trying to do a maximum entropy um model you're you're doing maximum entropy model on the reward function itself.

Um at the end of the day, what what ends up happening when you do this is this is why it's like basically just like active emperors. You get a reward function in a so you have some, you know.

organism or whatever and you're trying to do this for it and and it it's got some stationary distribution over actions and outcomes, right? It's inputs and outputs of a stationary distribution. That becomes your reward function. Like not directly, there's some math involved, but basically your reward function is a function of the steady state distributions over actions and outcomes.

So we could do this, right? We could take the current we could take the current manner in which humans are making decisions. And we could write down

Right. What's the stationary d what what is the current estimate of the stationary distribution of our actions and outcomes? So this would include things like everyone's getting you know, you know, this number of people are going hungry, this you know, and and you know, all the stats that describe like the inputs and outputs to our policy make you know, to our policy distribution.

Um and then we could just ask an AI Your reward function is the one that results in the same outcome that we currently have. Right, on average. And it would execute it and it would and and to the extent that it works, right, it it it it would it would ultimately result in a in an AI algorithm that just sort of is like mimicking human behavior, right? Or it's at least achieving the same outcome that we were achieving before.

Now here's the safe way to like improve the situation. You don't say endworld hunger, right? You perturb that distribution over outcomes, right? And just just over outcomes a little bit. And then you evaluate the consequence. Right. In the reward, in an empirically estimated reward function, right, rather than just sort of specifying one by hand, because that's the dangerous thing.