Making AI Accessible for Developers - JSJ 641

life that you're running? And you know, how will that fit into the life of somebody who's going, hey, I'm a JavaScript developer.

I remember GPT three somewhere I would say something even as symbol as ending a your prompt to get a completion off of with like a new line or something, it would like break the whole thing. So they'd be like, just don't do that, And like three point five was like, let's assume humans are humans and we will you know, make sure it works with whatever you give it. That

was pretty huge breakthrough. But even then you saw like all these companies rushing to add eight out of the product, or like we even have vcs who invested in us, Like every CEO is talking about redoing their whole roadmap to be focused on AI. People are rebranding their companies AI, and I was like, you know, I'm thinking of like crypto and all this stuff, and I'm like, no, we're we are not doing that at all. We are dabbling,

We're not changing significant plans. As we dabbled further, the potential, hypothetical potential became really clear. You know, if you work in that space of like you know, you've got a design and a program like FIGM on one end, and you've got a website or app that you probably have a mix of developers coding on because we focus kind of larger businesses generally, so you've got developers. You're not

like some small mom and pop shop or something. So you've got developers writing code, and you have people who are not developers trying to put out pages or update pages or something through a CMS. Where can AI help most? You know, we've seen all these cool demos and this is where things get confusing too, is there's a lot of demos that are not representative of the average users experience.

So you could, for instance, go into chat to BT and have it summarize a long piece of text, and every demo will do well and every user's experience will probably be pretty good at that. That's something that the elms are good at. Take a large amount of information and condense it down. And if you don't like the style of how it condensed it, you know, like the

language it used. Like when I paste a huge amount of stuff and I tell it like turn this into an email, it assumes that I'm like this corporate person emailing of a million people. No, no, we're startup for what are we now? Seventy people? I use a pretty chill vibe when I talk. Here's some examples of how I usually talk. Emulate that. GPT is still pretty bad at that, but Claude as much better in my experience. But anyway,

those are good. But then you see these other demos of like you know, these one off cherry pick things of like hey, I built me this whole program. That's awesome. In fact, Claude with artifacts is great at generating like a snake game or something. But that gives you a look into like what people would like to have happened. You know, people would like to If I'm a developer that has a Figma design coming from a design system

and it's got behavior implicit in it. This is a new dashboard mock up with we we've got APIs for this data, and we've got components for these charts, and it's you know, it's got a layout and Figma. If I can just turn that into like almost finished code to just start me off, there's always nuances I got

to do. But if you can connect it to APIs, assemble the components, create the layout and tail and whatever I'm using, and then let me work on it from there, that's pretty cool, especially when you know those things can actually work fairly reliably. That becomes like a why not type of thing? Why am I writing this all by hand? If the I can do that pretty effectively. On the flip side, if you're because part of what we are

is headless CMS. So if you're a user trying to create new pages within a next JS app or whatever, same thing, you've got this mock up of a page.

If I could just click a button and make that become real and then maybe use natural language to say, actually move the button over there, or when I click the button and should trigger the off flow or whatever, rather than learn this complicated tool of like, yes we have the off components registered in the tool, and yes we have our APIs connected over here, and if I know how to click one hundred buttons, I could do it. But if I could just say it and have it happen,

that's the pipe dream. That's the obvious reason to care. The question obviously is just how well can it do that? And then more importantly, how can we make a reliable path to make as many of those dreams come true as possible without being full of foot guns. And that's been kind of our focus in research and development of the last year or two.

So you can think of us as like the if you saw TL draws old make or not old, Like they're make real demos, you know they diagram you hit make real becomes real. You can think of us as like an entire make real application or platform. Most of the time those are in Figma designs already. Sometimes they're just in a gyro ticket or a slack conversation. We we imagine a future world is not too far where we could have like a slack bot where you tag

the builder slack bot. It looks at a thread, it summarizes your idea, implements, it sends you back a link. How does this look? Turn that to code, sink it to your code base, or just hit publish. Run it as a five percent test and see how well that does. Stuff like that, you know.

sign ups, some would do kind of cool things. But where I saw many fail completely were when they were adding AI for AI's sake, as opposed to solving major problems that only large language model school solve. So there are of ideas of things we could do with AI

for the sake of doing it. But one of the biggest problems that Pigma has in their code generation that we have and explorin this too, that site builders have is that people have a lot more ideas then they can get into code in a high quality, and any tool has tried to turn any figma idea into code in an automated way is bad. It's the simplest way to put it. They do a very very bad job. And we found a few techniques that work pretty well

that can only be done with llms. We found that we at the end of the day, like it's funny because I don't want to overhype lllms. I think they're a critical missing piece, but they are not the full solutions. What I mean by that is, you know, I've got some videos about like use AI as little as possible, and I still firmly believe in that most of our AI solutions that just look like, for instance, a Figma design is the input or a prompt and the output

is code. It looks like we just fed that into the LM and we got that out the other side a more code than that, and that's good because we have control over that. It makes the product differentiated. There's not just an enormous amount of code pre processing the heck out of everything, post processing the heck out everything.

But also we've trained our own models for specific parts too, and so you can take You could take various approaches here, but the one that's worked well for us is solve everything without AI as much as possible, break it down into the Tinese problems that are just not really solvable with just typical you know, conditions in code. For us, that was those were much smaller than you think, especially if you're really grind out the problem. It's really small.

But you can't do without AI. But there's probably gonna be some point, like if you're turning designs to code or prompts to code or whatever. That what you can't decipher with most models, you could train yourself. You know, you could do training as simple as things like decision trees, which can work great if you just don't know what value to put into a box or set of values decision, try training with like XG boosts or something. It could be phenomenal to wipe out a bunch of crappy conditional

code into one. Essentially, the AI takes example datas and kind of writes conditional code for you in a sense. Fantastic We use that for certain things. You can do fancy stuff like random forest. You know, a bunch of decision trees helping make a decision awesome. But what nothing can really do well outside of an LM is understand meaning of things and that's where again the LLM in our experience should not be treated as this opaque box where very raw inputs get out and very finished outputs

get out the other side. But when you start identifying, here's a basic example. You've got a responsive design and it's got desktop end mobile, and on desktop you've got a nav you know, with all the links horizontal, and on mobile we've got a Hamburger menu. There's no world where we could solve that without an LLM. But again, we're not passing the designs in and saying figure it out. We're not doing screenshot to code, which is kind of

awesome sometimes but always not quite right. People want you know you designed the position, you use the specific design tokens of Figma components, so those map to design tokens compotes in your code. When it's a screenshot, that's the conduit you lose all that information, and that's try not to lose that unless you're like, you know, again, you're just some random you're building some random project on the side.

Maybe you don't care, it's it's just beginning. But if you're doing Burker Company, you need a lot more than that for it to be useful. You don't want to have to get code and rewrite it all, and so you need to use some type of LLM to understand the meaning of that and say, oh, this should become an interactive Hamburger menu. It did not have to figure out all those other things, how components map, how design

tokens are map, all this stuff. We figured that out in advance, and we've found a format to pass that into the LM heavily preprocessed. You know, when all we're saying is here's baseline code that's almost done. It's just ugly. It's poorly named, it's poorly structured, and its might have some semantic misses, like the accessibility might not be quite right because we programmatically generated this with old school code and old school models, which are fast and reliable and

best to use when you can. But all you want to do is do some clean up of the code, refactor this, name the components better, give it some props, some better class names, and then anything that's meaningfully wrong, such as the way we converted the horizontal. Now maybe our default logic turns it just vertical, because that's kind

of a rule that works pretty well. Responsive design stuff side by side becomes vertical on the you know, narrow screen when that clearly didn't work well and you actually wanted a HAMBURGERM menu. Ho, the LM take a pass at that. Maybe that's the area that's least accurate, but still pretty much there, and then you're not having the developer at the end of the day have to tweak everything. They may just have to make some small tweaks to that Hambergerman. That's a bit rambly.

Where he basically breaks it down and says, yeah, so you have a like a gethub gethub ai chatbot, right, and so it knows about it's APIs and it's special things that it can do. And then you might have some other bot that knows kind of the next level up that orchestrates things, right, So he he was advocating too, to like break things down into really really granular things

and have it come together that way. Of course, his stack is kind of a stack of AIS and you kind of figure it out and set up context and things like that. But I imagine a lot of people and there are some comments to this effect on here too, and I think, I think this is kind of where I want to start with. What you've talked about is you know, Jack Harrington on Twitter said what are the

wrong ways to integrate AI and your application? And what are some of the right ways, And you kind of got into, you know, breaking the problem up and things like that. But then, yeah, Charles G said, AI is only reliable for prototyping and search. It feels like vaporware for most other stuff. It's that use models, you know, gotten off the hype train, and I think some of the stuff is going to get better. But yeah, so so you you talked about these specific instances, but what

what are the problems that it solves? Well, like before do I look at my stack and go okay, I want I wanted to hit here, and you know, maybe I'm going to hit you know, this and this and this and this and this and this, and I'll have different models or different l ms that I hit or whatever. But how do I know that this is a good fit? And then how do I start putting that in there? Because it does feel like some things are moving this way other things. Yeah, it's not there yet.

So first of all, I I you know, we will post that blog post here in the chat, so and I highly recommend for people to go and check that out. And and it also have to say that it also resonated a lot with me because of stuff that we are doing at the size sense, which is the company that I recently joined, which turns out has a very similar philosophy and how we're using AI in our own products. Uh, And maybe I'll touch on that after you kind of answer the questions.

result of how they're trained. They're just trained on lots of information of people saying you know, they're just saying things. You know, they're saying things as if they're.

times and answers and stuff like that. It's very annoying.

And so if it if it didn't always identify the dog as a dog, people would just kind of you know, as long as it was generally accurate, it was useful. And now it's it's problematic because it's generally accurate, but that's not good.

it's an essential part of a product. The product has a flow from start to finish, and the LM is wrong five percent of the time, that's a huge problem, especially when you use lms maybe for multiple steps.

But there are solutions to this. So actually there's two solutions that we found to be extremely effective. One has to do with actually that blog post you mentioned, Dan, and one has to do with some like microagent techniques. We've been really investing behind both in an open source project we've recently open sourced and some work we're doing internally in the product. So solution one that works fantastic. Let me give an example. Builder has lots of docks.

People don't know the answers to the questions, and it's tedious to try and come through all the docks to find your answers. No matter how we try and restructure the docks or surface the right information the right time, it's never good enough. And I know that when I use other people's products too.

large context windows is more effective than fine tunings. Rather than fine tuning with hundreds or thousands of examples, just fit ten examples in the large context and you'll outperform. But they didn't even mention, or maybe they did and I missed it.

So here's if you go to chat ChiPT and ask it, ask it a question like how do I add a user in builder do io via API? It'll tell you go to build a dio slash API, slash users send a post that API doesn't exist. There's not an API to add users to your account and builders through APIs, though it will tell you that it exists. That's a

huge problem. So then if you then try and augments this with okay, well we're going to take chatchipt or anthropicalaud APIs and we're going to supply our API documentation into it. Maybe we'll do some fancy based on embeddings and semantic search to find the right docs to include in that context and send it. It still will it'll get better, but it still will tell you to add a user. Go to API v one slash users and

so the thing that works extremely well. So what the elms are really good at in my experience is condensing information down. So if you tell it very clearly, here's a set of information. You are only allowed to answer questions using this information, and if that information has the answer, condense it down and provide the answer. If it does not, say I don't have an answer to this, it probably

doesn't exist. That works wildly well in our experience. Whether that's here's a lot of code, simplify the code, whether that's here's a transcript, tell me the key points discussed, whether that's here's the API documentation. Answer questions about the APIs. As long as you firmly say you could only use

this information and nothing else, it works. I mean almost one hundred percent, and I mean almost in terms of like LM still can surprise you from time to time, use a good one, So like don't use I hope peop aren't using GPT three or three point five anymore. For most things there's better options. But if you're using something like anthropoclout three point five's on it and you're giving it lots of information more than a needs, I'm telling it to reduce and only use that information. It

works great. Another example was so we have this assistant in our docks. It answers your questions and it works a lot better. Another thing that started doing, though, is making up links, like oh I added to the instructions at one point, like include links as much as possible, because a lot of what you're doing, or a lot of what we hope AI can help with the navigating docs is just knowing what for my use case, what docs I need to read, and so I told it

will the answer with links. Maybe just a way to quickly find the right links is a good, good example. But I started making up links. So what did I do? I added to the prompt. I took our site map essentially and some additional contexts on each link, and I fed into the prompt and said use links only these links, nothing else. And then fantastic. It always links to things

that are relevant. The more you do that, the more you tell it to do nothing more than cadets information and use nothing more than information provided, and just add a copious amount of information. Because these context windows are massive, it can work phenomenally well. So that's a use case that can be great. Another example of that is how our AI designed to code works is we have old school code, and old school models generate fairly accurate code

for a design. It's just verbose and ugly. It's just too much. It's one massive component of div soup and the classes are named like div one v two to three to four. If you take that and pass it to an LM and say, just reorganize this into multiple files and components, well named renamed classes according et cetera. Again, give it a model like the latest two point five son it or GPT four is pretty good, but song's

really good. It's just a fantastic job. And so we've spent an enormous at the time in the past trying to take generated code and try and make up class names. How the heck you're going to do that. That's back into that bucket of meaning. We don't know meaning in any type of code or model before at LLM really for this type of use case. But if you can

distill down that way, it works fantastically well. And then the last piece, the last piece we've learned here, and i'd love to touch on the agent piece because there's

one other technique that's great. The last piece we've learned is the user interface matters a lot too, and there's a user interface pattern that is just almost always wrong, yet we see almost everyone jump to all the time, including us, which is you want to assume that the AI is never perfect, and especially because when we're talking about meaning and summarization and stuff like that, it will not get everything right the first time. Even if it's accurate,

it may not be quite exactly what you want. So what you don't want is a UI like we used to have, which is like if I wanted to get builder to update my content in some way. It used to be like click a button, then you get a box, then you type in it and hit some of it, and the box goes away and it makes your update and it acts like you're done. Most likely you're not done. Most likely it got you closer, but you're not quite there at least some portion of the time we've actually

been landing on. A chat interface is almost always the right interface for an LM, even if your use case is not chat. So if your use case is imperfect but designed to code, and here's code, you should have a chat interface next to it saying what would you like to change about the code? Oh, I forgot to mention, I'm using tailwinds update it. There's too many components or

there's not enough components. You should be able to constantly iterate, and the chat format lets you maintain that context and never assume it has to be done the first time or what's coming soon for us is importing Figma designs and then like a Pigma prototype where you have like you know, you have this mockup of like clicking this

launches this modal and does this thing and updates this data. Well, we're going to suck that in and make real but not every You know, sigma is not a spec it is a suggestion, right, and so we need to treat it that way. And it's a description, So we're going to treat it that way. We're going to make some assumptions and then when actually you want clicking this to do that, you should be able to say it and see that happen. So that's one big bucket of learnings.

The other big bucket of learnings is if you want to use lllms on a loop, you know, you could describe an AI agents just an LLLM on a loop. Do a thing, analyze a thing, do the next thing, decide when you need to stop.

Slack example too. Here's our idea. Implement it because Builder has an API and could hook right up to your live app. You could have it. If it powers your homepage like it does for lots of customers like j Crew or Zapy or whatever, you could just tell it like hey, up to the homepage with this and they'll just go do it. Maybe it to be safe. Ruends it as a one percent test shows you in a day that the data is good, the ab test is winning some you know metric, and then you great scale

it up. You know stuff like that. Awesome, Just be be my homide that helps me do TV stuff that I didn't feel like doing. The problem though with agents that that have to take a sequence of actions is how they derail. Those small errors compound, so you need some type of mechanism to bring them back and analyze are they off track or not. And so this is

a technique we've been using. We call it like a microagent technique because it's about specialized and I think this is a topic that has to do with product development in general, which is, don't try and boil the ocean, like don't try and build Devin the world's do everything software engineer. Then suddenly you're solving everyone's problems simultaneously. It's

an impossible task. It's not a good idea. Rather, if you're an agent for one specific type of thing, starting there and then building up over time through feedback, through iteration, through all that stuff. That's how I always believe in building products. It's work better for me. So we have this open source project called Microagent, where the technique we've we realized works really well is if on each step

you can have something that's not AI. We're exploring if AI can be the step, but I haven't had good results yet. But something that's not AI can essentially test if that step was successful and if it was not, feed feedback in and then let the LM run again, so it has the feedback from the last iteration.

your problem if it doesn't work. Instead, it generates a test first, and it'll have all these input output examples of marked down to htmail and then it'll say it doesn't look good. You can give feedback on the test, like this is wrong, or add more or less or whatever, or just say okay. Then it'll it'll right the test and then it'll write code, and every time it writes code,

it'll run the test and any testing id feedback. It's a I TDD and it works really really well, especially for certain use cases that AI generally wasn't good at before, And so you get this sort of guarantee that with AI spitting out code, you don't have to any longer hope that that code works because it'll look good and it might work on one example but not others. The

test will ensure it works on all the examples. And when I run microagent and have it generate big complex things for me, I rest easy that if it passed all the tests and the tests are pretty thorough that it generates it works, and then if we find any issue subsequently, we'll update the tests accordingly. So it actually is a much more like high confidence solution. One other example is Figma has this concept called component sets, where it's it's like how it represents a button with all

kinds of variations, you know, like color primary. It's a design system feature exactly. It's how you have something similar to React opponents and Figma, so you can place the button and change the color from primary secondary, you know, aero state, true, you know whatever, it's cool. It's weird as hell how it works in Figma and Figma you actually design so if you have like three different props with three different options each. You have to make nine designs.

You have to manually code up or design every possible combination. It does us weird, it's funky, but it's it's effective. So how you translate that to code. What we do is we generate baseline code for every single variant and then we tell the LM consolidate down this all down into one piece of code with props you know, and react. And what we do is to verify the LM did

that correctly, we will run it through tests. So because we know what the end state of every combination of prop should look like, we then take what the LM provide it and test it against every end state, give it every combination of props and make sure that the essentially the snapshot is correct to the original spec and if it's not, we feed the feedback in and if

we've actually found you can play with it. You can either feed this to a slow high quality model like before three point five sonnet with anthropic there was claud Opus, which it was not obvious. I've had way a better experience with anthropic models in general than open A models. As becoming more popular now and more discussed but it was a little bit of a hot take in the past or less common knowledge. But anyway, you can play

with the knobs. You can either say we're going to do the big, expensive model and probably takes less iterations, or you can scale down to like we actually had good results with Anthropic Haiku their smallest model, and it would take four or five iterations, but that would run faster than the two and three on opus and you're still guaranteed an accurate result at the end. So if you have that automated check that can feed into the AI,

it could work really well. And that's kind of my point again about hopefully this illustrates good examples of a I can't be the entirety the brain of your product, but you can isolate these different techniques and use it to accomplish things that just would have been kind of impossible previously.

much more grounded and down to earth. And like I said, where I'm seeing very similar thing. At Sizense, we kind of build dashboards and stuff for BI and so in the past it was like, you know, you basically need to either best case scenario or dragon drop do you build your dashboard or you might you know, do some coding to build your dashboard. Well it's now we're working on, like you said, the sort of a chat thing where you can describe the dashboard that you that you like

and you get it. But at the end of the day, it uses all the components and know how in capabilities that we've already built for constructing dashboards. So it's so it's it's adding AI to existing infrastructure to make it better and to make it more approachable, I might say, and easier to configure and without you know, specialized knowledge.

more of the text boxer? I mean the demos and the demos kind of made me upset because you know, then you'd see people trying to do the demo where they were talking to GPT for and it wasn't working for them like the demo work. But you know, we're a lot closer. You know, how does this continue to advance?

of a ninety seven. And these things tend to slow down, though they also tend to be little s curves as well, Like I think one one not counter argument, but counterpoints to the idea of it slowing down completely is things like I don't know if you've used rock g RRO and I'm forgetting what spelling one is Elion mustling, but one is like a hardware company. A hardware company, I think it's jer Okay is super interesting because they can run Lama three at insane speeds. You type in their box,

you just get tons of text quickly. And they were one of the first to realize at a full ability to commercialized level, you can make hardware optimized for LMS and it's not just like a small percent better, it's like ten times faster and cheaper, which is crazy. And so we are not currently using Rock because we need larger context windows and LAMA three and all the others don't support the huge context windows that like Opening Eye and anthropic have I'm hoping, I know, medicinal training LAMAI

three four hundred B. I think it is. And so that, you know, maybe a magic there is a larger context window. I don't know that could be huge, but breakthrough innovations like that could accelerate things. One example of that is, like, hey, people love the idea of you know, as an example, because build has an API and SDK, it can dynamically

render your reaction app and components. When you drag and drop or use AI to modify it, you can see it all in real time, so it feels like Figma, but it's actually your react or quick or view or whatever app. So that's cool. And then changes can turn the code or published in their live to your users, so you can get people who are not even developers making changes, pushing updates, marketing pages or whatever you want. But what people really love the idea of is well,

why are we hard coding apps at all? Can't we make components and understand things about our users? And when I jump into you know, JCrew dot com, can it just dynamically produce an experience that's one to one, personalized and fitted for me. Now that's like Wow, that's cool. That's no, we're not even close to that. I've watched the code generate slowly enough times to say we're not

even remotely there. But then when things like Groc come out, which is probably the first commercially successful commerciuccessfulse and like I see people using it, I'm using it myself from time to time. Breakthrough an LM performance that was a big step towards that. You know, you see that paper of like what is it one point five bit YadA YadA for LMS, and it's like, hey, you might take

another le forward as well. I don't think Rock is using an architecture think used to something differently, so that might be a subsequent innovation to add on. Now it's like, okay, if we can do these in real time, maybe that is possible only for certain parts of your app. It's not you know, it's not like it's not like you deploy an empty repo and suddenly applications build themselves in

front of users in real time. But like you can start small, Like we were working with this very large company on the use case where they want to just be able to type in a very common use case. They want to type in a query and fetch the data associated to whatever that is. So like, I just want to see this data, and it can pull the data and visualize it for you. And the way they're

doing visualization is code generation. But just like any tool, this is very common, or they're using anthropic claude artifacts or tail draws make real, you always get just like funky, plain old raw HTML REACT. It's not like using your APIs your components, You're you're using tail whatever it's it's not using your it's not code. It's gonna go to production. It's all throw away code. But if you could instead be able to in real time, either offline or online,

assemble what you have your pieces. Offline use case would be like you know, I'm just gonna import a page to Figma and then I'll hit publish and then I'll go online when I'm done editing. So that's ultimately the offline use case. The online use case would be showing me these things on demand. So this large company that wants to surface your data, wants to generate code effectively

with your components, Well, what can do that? We already have these SDKs where you if they're aware of your components. They can dynamically render out in real time your stuff we're exploring, like, hey, yeah, you queer your stuff and you've it's aware of your just the components you want it to be for this use case charts, diagrams, pipe chart table, et cetera. I can just dynamically produce u eys based on your specific query in pretty real time

using LAMA three on rock. That actually works pretty well. So that's sort of like your first like online on demands generation and that makes sense for loged users maybe paying a certain amount per month, you know, that could be justified by the cost of serving that. But if we keep making these innovations on cost and speed, et cetera, you could get to a world where online generation of

parts or larger parts. I mean, let's take if you're Jay Crew, maybe you want to manually merchandise that hero. You want to promote this new product lines as the hero is this thing, but maybe down below whatever Amazon highly personalizes the products you see. What if you could just throw that at the LM and the whole UI is based on what you'd be interested in seeing. Those are kind of interesting. I can't remember if there was

a question here or what I was answering. But I want to throw that in because I think it's interesting direction that we might get to.

unless they have an infinite bank account. And similarly, you know, if it's not fast enough, then again, you know, it lowers the utility. So one of the things that I've talked to some people who are beginning to adopt AI features into their stuff is yeah, up to a certain point, it you know, it's it's great, and then it gets expensive. Right, So so how do you start to how do you start to manage some of that.

with your products really well. I don't imagine anybody doing that. Obviously, when we make open source AR projects, we have several Yes, you just supply your key and then it works that way. But yeah, those things really add up, and in a lot of cases, I think the companies are just trying to make sure that their bottom line is taken care of. They know if they don't charge twenty dollars a month, if you use this heavily, they might be underwater on you,

and that's a big problem. And a SaaS company usually wants to have eighty percent margins, So if you're going to cost.

put a value on various things. For instance, we just have our AI in our docs for free for anyone anyway can prompt and use it. In a lot of cases, you can one just ship it and see what it costs. Doesn't cost a ton, and two we actually know you know. Our platform is probably in the bucket of it's not trying to be like a fort seal, which is like it's one button and you never do anything. You never have to know anything. It's definitely there are things to learn and more power to get as you learn.

output is needed. And so that's how we cut down on cost traumatically as well use our own trained models when we can. If you are in a world where you know, Charles, you were mentioning the example of like you might use multiple llms, and we've done that in some cases. Like there's various ways you can do this, but there's certain use cases where you might want an LM to plan the work first and then another LM to execute on the plan. That can be a really

good one. It can be not expensive if you don't need to feed too much context in multiple times. The output of a plan is usually not a lot of tokens, but it can be. Let's say you needed a whole chain of lllms, or let's say you had a situation where LM one takes lots of context and provides a plan, and then you have different LMS executly each step with a plan. But they each need the large amount of context. Then it can start adding up and it's like, ah, you know, can we do that without LMS or can

we do other things? In the design to code flow, there's a lot of different steps and for us, some of the steps that can't be done with code I mentioned we train our own models. Those are it's hard to underscore how drastically faster, cheap, and more reliable training your own models than an LLM if it's a fit use case, and the fitting use cases might be more out outside of the box than you might think. Like, Yes,

one of our use cases is image detection. When you have a Pigma design, certain things that are like one hundred vectors should actually be one image when it gets to a web or app, and then the text around it should stay text and UIs and so that is a good sufficient use case for like an object detection model which uses a convolutional neural net very common. It's hot dog, not hot dog. It's identifying dog, but it's identifying an image from training data, which in our case,

you can generate that training data from the web. Scrape web pages, see what are images? Screenshot it, give it the screenshot plus the bounding boxes where the images work to. Now you've got infinite training data. You can train your models. But when it comes to things like decision trees, I think are way more interesting than people realize. You can break a lot of problems down to decision trees, where

you basically provide for a very specific problem. In our case, it can be things like here's a whole mess of layers and figma, which of these should be considered groups together, like you know, into a flex row or column. Things

like that. You can actually generate decision trees and be able to figure out or rain a forest, or you can get you can start simple like conditions and code, upgrade to desisionery if you need to upgrade to a rain of forest, if you need to upgrade to a neural network, if you need to upgrade to an LM at that point, and you can play the cost knob as much as you want. And there's definitely cases of products I see sometimes online where I'm like, the economics

are just not gonna work on that product. I know you're charging twenty month now, but it's not gonna last. You have to find one that will.

get hands on and build stuff. And still today, in my opinion, the way you'll learn more about how to build effective AI products with LLLMS, et cetera, is to build air practic with LMS and run into all the issues. I don't think the academics could tell me all the strengths and weaknesses for our type of customer, you case that an LLM. Willer once have if they haven't actually gone through it all. Now, if you look at like the research papers coming out, there actually are different categories

of research papers. Some are actually useful, some are literally sort of like the analysis of the usage of these things, like the one I mentioned comparing fine tuning to context models. That's actually useful, but that's not necessarily big brain stuff. Even if they have some weird calculus equations. You can feed it into. You could upload a chat to BT and ask what you want to ask and it'll tell you what you're looking for from the paper, which is nice.

But all the ways I have learned anything is just rapid iteration. So that applies to learning AI. That applies to building a company in general, building a product. That applies to learning how to market. I didn't know how to market anything. We made a ton of mistakes in the past. We still make mistakes now selling It's just I'll take it build and needed its first customer. What did I do? I asked around like how what is

sales like? And people like I don't know. You go talk to people and see if they want to buy. So I just start talking to people seeing what they want to buy. And I made a lot of mistakes, and now I could tell you a thousand things not to do, a thousand things to do instead, just from trial and error. And that's where I think a lot of people, whether they're learning to program, build more complex software, build, you know, get users for their software, whatever it is.

Two frequently people fail to just say I'm going to be bad at this, I'm gonna do it anyway, and I'm gonna do so many times that I'm going to learn infinite things not to do, which leaves a smaller and smaller window of what to do. And once I have that small window of what works and that long knowledge of what doesn't, at that point you could say I'm actually kind of good at that thing. And all it takes is doing And just like the agent's example,

it just means lots of feedback. So doing something and getting no feedback, you're not gonna learn anything. But I think like a big learning of mine is like, hey, you know, we'll be like one hundred person company by the end of the year. I've never run one hundred percent company. I think the biggest team I managed previously this company is eight people. So it's like, okay, how do I know what to do?

It's broken again. It doesn't look right now, but it works. Okay, now it doesn't work, but it's you know, you know, and you just you just do it. And so I just want to encourage people, like, if you're looking at this and you're going, boy, you know it sounds like kind of a slog and kind of a journey, that's

just the way it is, right. I will point out that, you know, I have a computer engineering degree, and so you know, I got into you know, writing code, and I still bang my head against the wall a whole bunch of times. I just had a little bit deeper foundation than maybe somebody else. But if I wanted to build the castle, all the all the all the head start I had was that I had a little bit

deeper foundation. I didn't have any of the walls up, I didn't have the boat dug none of that stuff, right, And so you know, you as you're looking at how you break it down, I just I really want to encourage people to just just look at this. I'm working on putting together a boot camp that you know, through a lot of this stuff, do it in October, right, And that's the same thing, right is you know, I

kind of want to be there. So when you run into the wall, you know, you're not stuck trying to figure out how to get off the wall and go do something. You know, do the next step. Right, you kind of get there faster, but you're still going to run into this even if you have somebody holding your hand. So yeah, just just be aware that that that's kind of the way that a lot of this goes. I kind of want to pivot this. You were talking about, you know, getting left behind right on some of this stuff,

or you know, it moves ahead quickly. And yeah, one of the questions that I get from people is, Okay, well do I have to learn it? And then the other question I get is because you're talking about specifically.

So when we talked to Obi Fernandez on Ruby Rogues, he was talking about the AI systems that replace you know, services like copywriting and add optimization and things like that, right, you know, aspects of running your business, right, and so you know, programmers, you know, would listen to that and not feel super threatened, right because they're gonna be in a place where they're going You know, that's not something

I do anyway. But you're talking specifically about, Hey, I'm going to take this figma and I'm going to have working code at the end, right, And yes, you've kind of let us know that there are various stages of effectiveness to this. But is it eventually going to get there where if I'm writing react or quick or something else that you know, my job is going to be you better write a really good prompt for the system

so that it'll give you the right code. And is it going to make it harder for people to get in? You know, is it going to cut salaries because the AI does a bunch of my work? I mean, how how vulnerable are we to this stuff?

gonna want both. If you're a developer and you're not playing with AI tools to see how it can help your workflow, I don't mean you have to be exhaustive. I mean, like you know, if you want to start with the basics, use chat Gypto. I personally recommend Claude instead. Cost the same and it does better, especially with code.

And if you're not using get a copilot or whatever most synonymous thing you have for your ide I would highly suggest you do those things because most importantly what they do is they build an intuition of what's AI good at and not and when it's out of suggesting.

There are certain things that I just know that AIS can do a good job on every time, and I have that intuition through repetition of just seeing it happen in real time with no effort for me, and I know things it will not do good at, and those things build a mental model. It's like training your brain, your neural network to know what the AI is good

at and not. So when you're building products with it or when you're trying to find productivity games for yourself, you have that intuition and where it's going to work and not. I see people every day with the wrong intuition, like, oh, can AI just do everything for me? It's like you have you used AI tools? No, No, it's not going to do that. That's going to go badly. So if you think about like how it affects jobs, let me

give you another example. Builder as a tool has always been a tool that takes the code and components you have and brings and takes work that you have as developer that generally you'll find tedious. And I think if you're thinking calmly and dispassionately. It's the tedious work you don't want to do, like marketing, wanting to move buttons around the homepage or change the color of this to see you know, and realize the test failed to they undo it. It doesn't feel good to be a developer

doing that. You don't want to be like a middle person in between, like say marketing and your homepage. That sucks. Objectively, it sucks, and so with builders. Sometimes people before adopting the product, they have a concern like, wait, is this going to take my job away? Is this? Do I have to be worried about this? I would say only one to five percent of people have that concern that CanCERN happens. I've heard it zero percent from real customers,

and I keep really close to our customers. Nobody's ever adopted the product that makes their development more efficient and actually worried about their job security Afterwards. When businesses can do more, they just want more. Their appetite grows faster.

of the job. Uh it's is that kind of what you're saying.

an atomic piece of the thing. And what it does for me that kind of makes my development better is that it gives me enough of the pieces that you know, from my experience, I can look at it and say that's what I want or that's close, right, and so from there then you know, maybe I hit tab and I have it drop it in and then I go and I modify it to be what I need, and you know, sometimes it's seventy five percent, sometimes it's one hundred percent, and sometimes it's no, that's just not what

I want. Right, you don't understand what I'm doing, But yeah, it totally opens those doors. And the other thing that I want to jump on here is you know you mentioned that, Yeah, if you have superpowered engineers, people are thinking that these companies are just going to cut their costs down to the bone and just have bare bones development AI generated code. That's generally not how companies work. Generally.

The way that they is they're going to spend what they spend to move forward with all the things that they think they want to provide to their customers. And so you're just going to be able to provide more of that.

minute and say, what am I paranoid against competitors? I paranoid that we're doing more faster than them, which means I want more superpowered aidevs who are more accelerated by the AI tools than my competitors, so we leave them in the dust. I'm not thinking, oh, we can do more, cut people. I'm saying I'm looking at we can do more, get ahead of the competition, maximize the resources we have, even hire more because now the ROI makes more sense.

I can hire another developer and pay them at developers out and get way more ROI than I did for that money. So it's an obvious investment that kind of has happened. As we got more productive, we started hiring more of these developers. And the way I would suggest thinking about it is here's an example. Has anybody ever hand coded a PDF? Heck no, You go to Chrome and you say exports pdf and it generates the PDF

for you. In my opinion, figma designs are similar. If you get to a point where you can generate it and then you can work with it from there, you're not gonna go back to hand doing it, it's just a waste of time. So the way you can think about your work is and I won't solve everything. You have a massive, multimillion line of code base, you have a very esoteric bug. You're gonna have to look into that and you're gonna have to actually dive into it.

A agents are so far from handling things like that. But when you're producing and getting ideas out and stuff like that, the drafting iteration phages can move so much faster if you can just suck in this dashboard and you see it. But you're like, I want to move this, I want to reconnect this. I'm gonna get on the code for this. I'm gonna get on the prompt for this.

You become like this orchestrator. And I think, you know, a lot of engineer don't like to step into a lot of engineers might like the idea of management because they like this idea of like parallel execution. You know, like you have people working under you, and so by you can be in charge of a project and you have ten engineers working on it, you can do theoretically ten times as much as if you were just the only one working on it. Cool but you realize people

are difficult. You spend your work not solving cool problems. You spend some the personnel issues. This person's like that person, This person is not convinced of the roadmap. And all this stuff happens and people realize, wait, I like building stuff. This is building stuff, and I get out of management. The best part of I could describe using AI effectively. It's like all the best parts of management. You have these minions or these tools generating things for you, but

without the downsides. They don't have any types of emotions about anything.

that you know that it speeds up. And I think one thing that a lot of people don't understand is that those iteration cycles happen now right, you know, And and if you've been on a project long enough, what typically happens is you'll get to a point where either you've accrued so much technical debt that it's impossible to work on or what what will happen is you'll you'll kind of get a couple of pieces in place, and then all of a sudden, everything else gets easier, right,

and so then the sky's the limit and and what the what you're saying is is a lot of the foundational pieces and a lot of the foundational thinking. You know that that's mostly just you just have to grind through it until you're done with it. That stuff goes away.

And so then all of a sudden, it's we have these greater capabilities, these greater opportunities, and so we can instead of the grind taking two months, the grind takes two weeks because we get that much further ahead, and then we can modify what we got from the AI or work with the AI for the two weeks to get where we needed to go, and then we can turn around, and that iteration cycle happens faster exactly.

totally agree with that, But it's still it. There's still the question of how knowledgeable do I need to be at AI? Because there's a difference between being able to, I don't know, drive a car and be the person who can pop the hood and start fiddling with the stuff there. So how knowledgeable do I need to be about AI in order to be effective as a developer, to be desirable as a developer in the upcoming one, two, five, ten years.

how well at auto complete certain docks. Let's say you make a writing products well by learning how copi has worked well or not, or Tatchubet's work well or not. And now suddenly you work on some type of writing product, a Google Docs thing, and the company wants you to build some type of AI feature. Well, you already have some baseline knowledge and intuition built around this. So the two categories probably are one. Using AI tools to be

a more effective developer. You should always try and be more effective because if you're not, your peers are, And in a certain sense, you don't want to be just so behind the train. Everybody's at a whole new level than you because you had just hopped on the train. You don't want that to happen. You want to be using the best tools, the best ability to be as good as your peers, be one of the best of your peers. On the other side, there's building AI products

that is shockingly similar. And if you really want to be if you think in five years every product will be an AI product, I think that's an extreme thinking, but you could think of it differently, as more products will have more AI features, and if my company might want me to work on it, it's nice to know how to work on it. Again, I don't think the way to do that is to go read a bunch of white papers, build your neural network from scratch, understand

how transformer works into the hood. Again, I think that's like trying to solve psychology problems with physics and a knowledge of F equals M. It's not going to get you that far. Solve psychology problems by learning about psychology and practicing psychology in some form, studying it directly and learning that way. And I think that the beautiful thing about AI is it can solve your own problems. You know, we talk about like the types of work dev don't

want to do. You have this really exciting opportunity. Let's say I think we've talked about code debt, like, could you make an AI workflow to help you with managing code debt, refactoring, cleaning, as cetther code. That's a problem that you right now, as an engineer, might say I hate dealing with this. I can play around with AI to try and solve my own problem. Engineers love over automating things. You know, let's spend a week automating something

that could have taken a few hours of front work. Everybody, everybody's guilty of it. If you're probably any good, you probably this is how you think.

features give Copilot with some nuances. So no, I don't think you need to waste tons of time adopting every tool. There is a learning curve, but just adopt the basics and use your own time to do your own side products sometimes to use AI to solve your own problems. I think that'll make you equiped to be a very future proof, very capable developer. One years, five years, ten years from now.

on your machine. Honestly, if weren't building a products, I wouldn't know why people care that much about that. Sure, cost is a thing, and so we start exploring it, and we tend to find that the local models aren't powerful enough yet to solve these use cases without computer.

it's good enough. And that's where getting your hands on is the best way to know those types of things.

And so you know, the more you can get into it ahead of the curve, I think, the better off you're going to be. So thanks for coming, Steve, Thanks for having Yeah, we're going to do our picks. Then we're going to wrap up. Dan, do you have some picks for us?

including one child and one baby. And it's a situation that I hoped would have been longer and still hasn't been, and hopefully it will be. What can I say, It's going to be very, very difficult to even look this person in the eye let's put it this way. It's it's I'm it's it's it's just such a statues situation. Anyway. Sorry for bringing in such a bummer, but that's the only thing that I really had that I wanted to mention. Uh So, so yeah, hopefully you can pick up the mood.

president or who ever, to help figure some of this out. Yeah, We've had some stuff going on in this country too, but I'm not going to go into it.

this week to play a board game. And this is one my So my nephew's here from Illinois and his his parents are going and doing job interviews in Wyoming, and so they dropped him off here in Utah instead of dragging him around Wyoming and having him be bored with them. He can play with my kids. I say, play with my kids like he's five, he's seventeen. But the game I'm gonna pick is Mysterium. I always do a board game kicks pick Steve at the beginning of

my picks. And so Mysterium is a game where you have one person giving clues and then you have other people trying to figure out what the clues mean. And so you have all the players but one are psychics. And then the person giving the clues is a ghost. And there have been murders in this house, and so there they hand cards to the psychics, and the psychics try and determine, you know, who the person, the place,

or the murder weapon is. And then there are expansions where instead of doing a murder weapon, you do a motive. But it's it's a pretty fun game. Takes an hour or so to play. Sometimes it's really hard because you know, you've got all kinds of interesting things on this card. It's just this, you know, this picture, and so anyway, Yeah,

you go through the rounds. You pick the first person first, you guess the place next, and then the object, and if you get all three and everybody gets if all the psychics solve their murder, then the ghost has one final round where they give three cards and depending on how well you were doing it guessing whether or not the other psychics were correct or not, and how early you got yours done, you get to see one, two, or three of those cards, and those are hints toward

one of the murders that the psychics solved, and that's the ghost, the person who's beginning to concluse that's their murder. And you know, it's kind of a majority wins thing on that one. It has a board game geek weight of one point nine. I keep telling people that kind of the average you know, friendly board game that's approachable to people who don't play board games is about a two, right, and so this is right in there in that area. It's kind of a fun social game, and I really

enjoy it. The kids were playing it. They helped my eight year old play it. I don't know if an eight year old could pick up on all the nuances of it without help, but they just helped to play. And so anyway, fun game. Like I said, there are expansions for it. It came out twenty fifteen, so I'm gonna pick mysterium. I was talking to Dan before Steve got on. He asked me how I was, and I was like, tired because I had just gone for a run. I'm training for another marathon, and so I'm going to

just do a couple of shout outs on that. I have been doing the trainings off of a Training Peaks training that I bought. So Training Peaks is free, and you can buy training that you can stick on your calendar and those are just fixed costs. So I think

I bought this one for twenty or thirty bucks. And so you just tell it when your race is and it, you know, Mind's what a sixteen week program, so you know, I had to figure out when it started and just stick it in and it puts the workouts on my phone on my garment app, which sinks to my watch. I have a Garment four Runner two thirty five, which is not anywhere near the nearest newest model, but it works great. So anyway, that's what I'm doing for running.

And then finally this week, I'm working on getting AI for JavaScript dot com up. I'm also doing AI for Ruby dot com. And so you can go and you can get on the email list, and I'm going to be emailing out, hey, this is what I'm doing with AI this week. These are the APIs I used. I plan on doing all my examples in both languages, so you'll get the JavaScript ones on the JavaScript list and the Ruby ones on the Ruby list. And I'm also looking at putting together summit at the end of September

or the end of August. Beginning of September. We'll probably have people like Steve. I've been talking to a bunch of the other folks in the Ruby community that I know that are doing AI, and really I want to be hitting it at this level right where it's it's not hey, here's how you math your way into models

that work. This is hey, you've got a model that works, or you know, here's how maybe you modify a model that you have to train it a little bit, but I want it to be approachable for people who want to add AI features too applications, not how to solve whatever thing by you know, managing a data lake and then feeding it into a system that generates the model and then how to test the model and all that stuff.

Or we'll get into some of that at a high level, but mostly we're going to be talking about, Hey, here are the APIs for Claude or GPT four or mid Journey or you know whatever. You know, I do podcasts, so Whisper, you know, So here's how you use these to get what you need. And then you know, here's maybe how you tie a few of them together to get a more complicated result than you want. Then you can get from any one of them. And so that's what I'm looking at. And so anyway, I'll be emailing

people on the email list right now. I'm just finalizing my system because I've kind of had to rework bits of the email stuff that I've been doing. So if you're interested, go to AI for JavaScript dot com. I'm also going to be doing weekly calls for JavaScript geniuses. You can find that at JavaScript Geniuses dot com and that's going to be sort of a back and forth,

ask questions, get answers. If you have feedback or ideas for people, you can help out, you know, and help people get where they want to go with that as well. But kind of more of a mastermind or group coaching sort of setup is kind of a blend. But we'll be doing calls for that every week, and you'll be getting some AI stuff in there too. So anyway, are my picks and myself promos? Steve, what are your picks?

I've used. Obviously, I'm biased here obviously, but it kind of solves some of the fundamental challenges. I've got a blog post talking about this in better detail, but some of the challenges of agents, and I think that I may.

to a lot of interesting things. So people trying it and giving feedback I think is really interesting because I think that's how we solve these problems over time, is how to make agents work. Make it work really well for something, And I think this tool works really well for a certain type of problem. And I think I have ideas and experiences that suggest what's really good at But I'd love to get more feedback from people what they've found success and not success with, and just more

input gehab issues, poll requests. People have added some cool new features to it. So I think it's an interesting area of both a practical tool and like an avenue for research and development that you could be part of in AI and AI agents and stuff like that that I would value. I want the project to be as community driven as possible, so check it out, give feedback great issues, send pull requests for improvements or fixes or whatever, and let me know what you think.