Inside image generation’s Renaissance moment - Episode 19

Hello, I'm Andrew Main, and this is the OpenAI Podcast. On today's episode, we're talking about Images 2.0 with researcher Kenji Hada and product lead Adele Lee. They'll discuss why the new model represents such a major leap forward. the evaluations that mattered most during development and what people are creating with it now that it's widely available.

If Dolly was the Stone Ages, Image N 2.0 is the Renaissance. It's not only great artistically and aesthetically, but it also incorporates, you know, science, art, architecture, all in one image.

We looked at it and we were like, all right, this is better than images.

Uh Adele, tell me a little bit about how you became a product manager here.

So I joined OpenAI a little over two years ago. And before OpenAI, I was an investor my entire career.

Oh wow.

So I was in private equity and spent three years at Vet Red Point Ventures investing in AI and software companies. And when I first joined Open AI, it was for a completely different role. I was thinking about how do we build out our data and compute infrastructure. And over time made my way over to the product side and for the last six months have been working on Image Done.

It it's it's interesting how you sty yourself going from one world and finding yourself into this space here, which is kind of cool, you know, to think about the idea that you have this sort of, you know, ability to be useful in different ways.

Absolutely. And I think the role of a product manager is just to do the job that needs to be done, no matter what it is. And for Imogen in particular, it's been really awesome to flex a lot of different muscles when it comes to

Kenji, how did you end up working on images?

Uh actually like when I first started at OpenAI, I also started about two years ago. Um, I was working on like some random audio project initially, just was my first project. And then at the time I just found my way just working on helping them work on uh images one point oh the prior to the launch. Um And so gradually I moved more and more onto the project and then I became full time on it basically.

What has the reception been like right now for the moment?

model. In the last two weeks since we launched the model, usage is up more than fifty percent. More than one point five billion images are generated every week on Chat GPT. And we've seen viral trends emerge across the world. um all the way from trends in Asia for color analysis and stickers to US for crayon and scribble or going viral. Um, but also a lot of people exploring emergent use cases. And I think it shows the dynamic range of the model, but also how people

This felt like a really big shift, almost worthy of me being not even being a an images too, but almost like just a new paradigm because just the capabilities are through the room. What made that possible?

When we started working on this project, I think we sat down and we discussed What is the step change of capability and use cases that we wanted to build towards? Um, and we believe that image generation has the ability to do so much more than it what it does today. You could distill every single output uh or visual content that you see today into an image. And so that was the mandate that we sought out to improve. Um, and with this 2.0 model, we've improved on various different dimensions.

We often listen back, uh listen to feedback on social media too. So we kind of just take all these things and basically are just aware of it and try to make sure that they're mitigated or completely fixed in some cases in in the next iteration.

What kind of use cases are you seeing? What are you seeing people do with this now?

I think one that's particularly close to like the research team as a general is like infographics, tech. Um, I think text in images is like so much better nowadays. So um I think it just opens up a lot more productive use cases. And at from like the research side, we th kind of think, you know.

So you mentioned text. I remember the early models, uh no disrespect to chimpanzees, but getting into the spell like opening I even looked like a chimp did it. And then now I'm looking at pages of text and finely detailed stuff. And I know that as models get smarter, variable binding, the ability to put things next to each other improves, but this was just a big improvement.

Yeah. But I don't think it's like completely unexpected. I think you you see a lot of growth in between uh well, first you see between Doll E three and you know GPD images one. There was uh If you ask for a grid of random objects, You go from maybe like five to eight in Dolly three to maybe around sixteen in images one. And then with one point five.

That was a test I used to use for like the really old models back with like Ada, Babbage and Curry, like list a hundred science fiction books and then some of'em would get by the time I got to like twenty two would just start repeating stuff. So we've seen stuff too, like 360, 360 degree panoramas. How did that happen? Yeah.

That really came from the emerging capability of the model, which is the ability to render images in any aspect ratio. We discovered that people were generating really long, amazing panoramics, you know, skinny bookmarks as well. And one of the cool capabilities of the model is that not only were you able to generate images in this panoramic aspect ratio.

Thing I did was I made a version of dogs playing poker. I put that in there so you could sit there like you're one of the dogs looking around in there, which was not something I expected, but it's it's

It's fun. Yeah. I mean it's really awesome to see how people are exploring new use cases and fun things that they're creating with the model, um, even far beyond what we expected users to be using it for. think when we were designing the model, um, we were really deliberate in understanding what people really wanted to see from image generation.

It's funny too how one of the things that was trending was taking popular images or photos of people and then having the model make like kind of janky looking Microsoft paint versions.

Yes.

And did you think that was something you would see was that people are going to use this incredibly capable tool to then go make, you know, these silly looking things?

Yeah, it's funny'cause it takes a lot of intelligence to actually create something that is imperfect.

That's what I tell people all the time.

Yeah. And it's definitely very interesting in the viral trends that we're seeing online right now. Um, one thing that I think people are really striving for is authenticity, imperfection, nostalgia. We're seeing that in the MS Paint prompt, crayons, um, all different kinds of generations that people are creating. And that really feels like the theme of consumers is

Kenji, was there a moment with this model where you're saying to yourself, wow, I think Think this is ready to go.

You know, as it's training, we take a checkpoint and then like we just sample from it, right? And just see, okay, how good is this thing? And I think like We just sampled them, checkpo a model uh uh an image and

All right, this is better than images one. Okay.

I remember watching the iteration of one of the early versions of Dolly and how at first it was sort of the wispy, sort of weird, sort of the tendril sort of thing, and talking to one of the researchers like. Is is that gonna go away? It's like I think two, probably two runs away from that. And then just like that. The ability to predict that was amazing to me, and all of a sudden everything got crisp and clear. Yeah. And then also like looking at

Yeah. I mean it was just very obvious. You just you just take the early checkpoint, you just sample an image from it, and then you just sample an image from uh, you know, images one and you just look at the two and you're just

There's no why don't I like this garbage? This is a

But what the image was, it might have just been like a picture of like a woman at a sea on the seaside, like yeah, you know, overlooking a seaside. We just looked at it and we're like, All right, this is like no no question.

That was the big the big the big jump was the photorealism of going from something that looked that was more of a a glossy idealized magazine cover to something that looked like a really good photograph. Help me understand, like besides just more compute, how did this happen? How did you get a model that's much better and also that doesn't take an hour to generate an image? The times are still I I remember

I think we learned a lot uh in each release, like between one, one point five, now two. And so we take each each of the learnings that we've made and we've, you know Like for example, speed, right? Um, you know, one of the things is like, oh, can we make the model more token efficient or or or something like that? And uh, you know, we did a lot of work to make it to make it put produce very good images with what

I think the post-training for this model was very interesting in the sense that uh we really had to think about not only does the model understand world knowledge and how things look and you know, science concepts, um, math, et cetera, in an image.

Do you have any personal favorite benchmark tests you like to do? Things where you say, I want to see it make an image of this.

I have a eval that I call the Me Me Me Eval. Okay. It's essentially a hundred photos of myself.

Yeah.

And my friends and my family. Um and I put everyone in goofy positions. I have about a card or birthday for every single person. Um and I think it's a really great eval in the sense that uh you only know the people around your, you know, faces the best. Um, you also want to create funny things with the model and th do things that are relevant. And so one thing for me as a product manager um that I'm testing is not only

How about you?

Besides the grid one I mentioned earlier, that's probably the one I've used the most. For a while, I think Dibya and I were doing a lot about photorealism. We were trying in real hard to push on that. Um uh just basically I know Divya's favorite one was like A woman holding an or a jug of orange juice.

Yeah.

There's like so many images of a woman holding a jug of orange juice.

Well I actually feel like the researchers had a more standard set of images like than they like to lead on.

And you get like the standard can it do somebody writing with their left hand and a watch on their right hand and a clock showing this. I think the big the big leap of the image is like probably one or one point five was like a half full glass of wine.

The wine glass folded the room.

Yeah, full of the brim. Yeah, it was exactly. And there were ways I was able to prompt it to do it, but it was oh it was really had to get a really descriptive like, you know, red liquid inside this. This one is so fun to prompt. There was a thing people said, oh, can it do like, you know, can it do like pixel accurate pixel image style art? And somebody was like, no, it can't. And I, when I hear that, I'm like, okay, let's try. And I found out if I gave it like a

People come to ImageShine with very vague prompts.

Yeah.

Make it better, make me look better, um, you know, make me cuter. All these things are really vague. And I think it's really the job of the model and the harness to distill that into actually what users want. And I think that's a personality of the model that we've trained over time that we've really harnessed the power for. Um, and honestly, I think it also yields a lot of really surprising results that people may not expect. And that surprise is just part of the fun of using Imogen.

I've seen like two kinds of prompting sort of emerge. And I I remember back with Dolly, I thought like, Oh, I'm a prompt engineer. I'll be great at this. Like I'll be really good at this. And I and I'm you know, I'd make a raccoon in space, I'd be like, feel proud. And then I'd see an artist.

Definitely. I mean, we work with a group of artists um very closely when we develop this model. And we're very inspired by artists, designers, marketers, all these different professions that I think have a different way of approaching their profession. And one of the things that was very important for us is we wanted to take the inspiration as well as the best practices for those professions.

Well it it's it's interesting because I think that a lot of people worry that oh, I just push in a button, I get something beautiful and and Each each model that gets better. It's easier, as you said, to not have to put a lot of effort into it. But when people do put effort into it, they are getting even more amazing results. And it seems like actually the

Yeah, definitely. I think Having creative direction or taste or judgment and bringing that to the model is the best way to push it further. I think one thing about this model that I'm really excited about is how it expands the creative outlet for people.

The ability to do great infographics and diagrams is very powerful. What kind of feedback have you been getting from people in research and education?

We actually have an internal alpha channel um where we test our models. Uh and in that there's like a sub channel dire dedicated specifically towards educators of any um level. elementary school students all the way up to graduate level. One of the coolest things I saw was there was a biology professor and he Put like these graduate level textbook rendering pages of things I had no clue about. And he said it was perfectly accurate.

I think the ability for this model to distill very complex topics into something that is really easy to understand within an image is one of its strongest capabilities. And we've seen this with students, with teachers who are using Imogen to learn different concepts.

I remember when I was in school and kind of prior to a lot of kind of multimedia blowing up, posters were a big thing, classroom posters explaining stuff. This really reminded me of how powerful an infographic can be because It allows you to bring as much attention as you want to it. Yeah. And you can spend the time looking at it and seeing it and you can put a lot more detail into it.

I think one really awesome visual shift that I've seen with Imogen is that now in internal presentations, over fifty percent of the slides are created with ImageGen. Wow. Um and that permeation of communication via images is so powerful when you're trying to explain your concepts or illustrate what you mean. And I think infographics and the text rendering capability, as well as the composition of the text on the page, is incredibly powerful with this model.

How do you see the progression of this? This is great, but typically anytime I talk to somebody opening eye about what they're working on, they're like, yeah, this is good. But

I think we're still super early in exploring all the different use cases that people are really trying to push the model with. And so one of the things that we're really excited about is what is that next?

I'll tell you another thing that was kind of amazing. It was like um all right, books. And so like every now and I have a book come out, I've got to change my social media headers. And I just went and I said, oh, find my book cover and write, you know, create a a post, you know, create an appropriate size social media header that I can put on X or Facebook or whatever. Like, let's say first shot, first shot, right aspect ratio, everything.

We basically did that from the start or trained the models to be good at that from the start. I remember like I worked on the initial D Risks of of ev basically it could do any Aspect ratio that you ask? Yeah.

Yeah. Yeah. You can now um really just easily specify the outcome that you want. Yeah. Like in the case of yourself. You're like, I want promotional material. I don't have an idea. I didn't specify exactly what I wanted. But the model was able to do the research and then give it to you in the style and aspect ratio that was relevant to you. And that's super powerful. We're already seeing this. Um you know, y you're you're an author.

This does feel like the I think it feels like the first time where anything I can reasonably come up with that does a pretty good job of it.

We think it's a new paradigm for any generation altogether. You know, we set this in the launch video if Dolly was a Stone Ages, Image N two point O is the Renaissance. Yeah. Um, and I think that is so true because the model, it's not only great artistically and aesthetically. But it also incorporates, you know, science, art, architecture, all in one image together. And I think that composition um and knowledge that the model has just means that the outputs are so much more trustworthy.

Yeah, I asked it in Codex. I said, I took my website, I said, could you make me, like I had the image in, could you create me some, you know, some different concepts for it? And it did these contact sheets. I asked for contact sheets to that, give me like four images there. And I said, Oh, the one on the upper right, can you go make that? And I watched Codex go make that, which was like, this feels like magic. And then uh they've implemented as part of pets.

Yeah. Sprite sheets are going viral. Yeah. Um same with game design. People are loving using image design to help them create new worlds.

Any any hints on how to do better sprite sheets? Yeah.

I mean I I've tried to make you know GIFs internally and I think if I just use like the thinking mode or codex and you basically just ask it to generate one initial sprite, it's really good. And then you can just say, can you make the

The consistency across multi-images has been amazing. Um, we've seen a lot of people try creating 10-page comic books with consistent storylines. um, you know, multi-page slides. I think that consistency of characters and aesthetics is completely unique to this model.

That was an example too where there were a lot of workflows out there for working with image models that uh you had that were kind of janky, but you had to figure out how to do. And it's great now because I can do stuff where I can like create characters and say make a character sheet with the different poses and stuff and just go feed it back in and say, Okay, now doing this, now doing that, now doing that. And that's just such a it often

Yeah, for sure. And we're still trying to improve that as well. It's not perfect today. We're really trying to develop this visual creation layer for people because every single person you have an aesthetic or personal style or preference. And we're really trying to imbue that into the product that we're building so that people can get to the output that they're wanting easier and faster with Image N.

Any any parting prompt tips for people?

Well, one of the things I would suggest people try is ImageGen thinking. So if you navigate to the thinking or pro models, we have a more powerful version of ImageGen in that experience. And in that model, you actually are able to search the web, analyze files, leverage tools under the hood, which then yields a better quality and higher composition photo. And the suggestion that I have for prompting that experience is

I think just being very particular about the style or like what you like in general. Like for me I like minimalist infographics. Sometimes I think the model can be a little dense. And so I just Maybe I'm just a simplistic kind of guy. So I just like very thin very clean, a very clean look. So I like that.

Adele, Kenji, thank you very much.