Open source AI to tackle your backlog

And then we have, Graham Newbig, who is co founder and chief scientist at All Hands AI and associate professor at Carnegie Mellon. How's it going, guys?

If you're using Vibe coding tools, you know, what impact is that having on on code quality? All all of these sorts of things. So I'm wondering, you know, from the perspective of, you know, all hands and the work that you all are doing, what does the kind of environment around these code assistant, vibe coding tools look like from your perspective right now? Kinda what does that ecosystem look like? And then maybe set all hands in the in the context of that would would be helpful.

One is on the one hand, you have a lot of tools that are really meant for, like, rapid prototyping. There's some really cool stuff happening there, stuff like Lovable, Bolt dot dev, v zero dot dev, stuff. They tend to be very visual. You're getting, like, quick prototypes of games or websites, things like that. Some really fun stuff happening there.

And the stuff that's enabling, like, a whole new set of people to experiment with software development, people who, you know, maybe, like, designers or product managers don't really have coding experience, maybe have, like, very little coding experience, they can now build whole apps, which is super cool. And then on the other end of the spectrum, have stuff that is much more oriented towards, like, senior developers who are shipping production code. They're working on a on a code base that's gonna go and serve millions of users, where you have be a little bit more careful about what's going on. Also, really cool stuff happening on that on that end of the spectrum. And then the other way I like to to bifurcate this space is that you have some tools that are very tactical.

So stuff like GitHub Copilot, where it's, you know, inside your IDE, you know, it's it's suggesting code, like, exactly where your cursor is inside the code base. You're, like, you're zeroed in on a task, and the AI is just, like, helping you move faster through that task. And then on the other end, you have these tools that are much more agentic. Right? They're able to just take a quick human description of a problem, and then go off and work for five, ten, fifteen minutes while you go get a cup of coffee or work on a different problem or catch up on email, and then it comes back to you later with, with the solution.

And open hand sits basically on the right end of both of those spectrums. Right? We are really oriented towards senior engineers who are working on production grade code bases, and we're really oriented towards this more agentic workflows where you're giving an agent something to work on, and it can iterate forward on its own without you having to babysit it, without you having to be, you know, you know, squinting at your computer screen trying to figure out, you know, where you should be editing.

One somewhat surprising thing is how ineffective this paradigm ended up being from two perspectives. So the first perspective is it didn't really and this is specifically for the case of software engineering. There might be other cases where this would be useful. But the first is in terms of effectiveness, we found that having a single agent that just has all of the necessary context, it has the ability to write code, use a web browser to gather information and execute code. Ends up being able to do a pretty large swath of tasks without a lot of specific tooling and structuring around the problems.

And then the other thing is building many, many different agents is kind of a relatively large maintenance burden if they're not very easy to define. So we've basically gone of full in on having a single agent that can do many, many different things. But in order to do that, it has to have the ability to pull in whatever information it needs. So we have a framework called micro agents where basically you can pull in a new prompt or a new tool or something like that for a particular task, but the underlying overall agent is a single agent that can do many different things.

Instead of just completing a class name, now it's writing like, you know, several lines of code. So that was like a huge boost to my productivity when I adopted Copilot. Was like, yeah, this is amazing. And then I was still, you know, for bigger chunks of code, was like going to ChatGPT, and I was like, hey, can you write a SQL statement that'll do x, y, and z? Or, you know, things like that.

And often, I found myself doing this workflow where I would ask ChatGPT or Claude to, like, generate some code. I'd paste that into my IDE, run it, get an error message back, paste the error message back into into Claude or ChatGPT, and I just do this loop. And at some point, was like, well, this is dumb. Like, I'm just shuffling text between one app and another. And that was actually when I I built my first, like, agent basically, where I built a little CLI that would just do that loop with with Anthropic on in the background.

And that's that's kind of like the core of what an agent is. It's it's doing a full a full loop where you basically you give a problem to the agent and say, yeah, like, okay. I wanna write a SQL statement that does x, or I wanna modify my app to add this new feature. The agent writes some code. It runs the code.

It sees what happens. It gets some kind of output from the real world, whether that's like the output of a command or maybe, you know, the the contents of a web page or the contents of a file, puts that back into the LLM's context, and then it can take one step forward closer to its goal. And then, you know, you can, as as you get kind of better and better and more accurate at taking one step closer to your goal, you can take on longer and longer range tasks. So I would say in the beginning, agents were really good for things that would take like 10 steps, you know, something really simple, like implement a new test and then make sure it passes. And now they can implement, you know, things that take hundreds of steps, which is really cool.

I mean, that's that's the changes that we've seen over the last, you know, six to twelve months is that they're able to take on these huge tasks. So I can say, implement feature x, you know, front end, back end, and add testing. And today's agents are able to just continue executing, stepping forward into that until it comes to full PR where all the tests are passing and it's just kind of packaged up and ready to go.

And what does that persona look like?

I would say one of the kind of failure patterns I see with the technology is a lot of times a junior a junior engineer or somebody who doesn't really know how to code, you know, Vibe codes their way to, like, a pretty good MVP because these agents are especially good at, like, Greenfield stuff. Right? They can build a, you know, a to do list app all day. And then as you layer on more features over the course of, like, weeks or months, the code base just starts to rot a bit. Like, the agent adds a bunch of maybe it, like, duplicates a whole function because it couldn't find the original function, or it just keeps expanding the single function so that it's like thousands of lines of code and has all these forking paths.

If you don't have somebody looking at the changes that are being proposed and critiquing them and like telling the agent, hey, you you added this new function, but we have an existing function that does that, or, you know, this function's getting too big, please refactor it. If you're not looking over its shoulder and critiquing its work, the code base will just grow into this monster, and you'll have to throw it all away because it's just it's beyond repair.

And we believe in the latter approach basically. We believe that if this is going to have a big effect on software development, software developers should be able to participate in that. That's kind of the ideological point of view. The other point of view is we also believe that from a research perspective, open source, especially from the point of view of agent frameworks, not necessarily the underlying foundation models, but from the point of view of agent frameworks, open source is not ever really going to be behind the closed options. The reason why is because academia and all of these people really love this topic.

They really love working on it. If we have an open framework and we can provide a platform both from the point of view of having a good code base that's easy to experiment with and providing resources to people who want to do experimentation on these topics, then the open source community together will be just as good as any company that is working on this in a closed manner. And so instead of reinventing the wheel, we can all invent it together and come up with something really good that's good for developers, interesting for the academic community and other stuff like that.

We're still working through the best way to fix that. But especially front end stuff, we have a new front end capability that we'd like to have. We've had a lot of people join successfully through that. Then we've had longer term research projects where we collaborate together with people in universities And we've been pretty successful at doing some interesting things there, I think.

Could you just talk us through those options, how kind of people can access this and you know, what they'll see, how they how they integrate or how they connect their code into into all hands to get started, that kinda getting started picture.

You just run Docker run, and you'll see, you know, a web interface running at local host 3,000, and you can just drop in a prompt to the agent. You can also connect it to GitHub by generating a token inside of your GitHub settings, plugging that into the UI, and then you can start to pull and push to your repositories. It's a little bit tricky running things locally because not only do we run the application in Docker, but when you start a new conversation with the agent, we want to make sure the agent's work is done in a nice sandbox way so the agent gets its own Docker container to work inside of. So there's a little bit of trickiness we have to deal with a lot of, like, troubleshooting, you know, why isn't Docker behaving properly kind of stuff. So it's it's it's a little bit of a difficult application to run locally.

So we actually created app.allhands.dev where you can use OpenHands in the cloud. And this is a really just like it's pretty much, you know, one for one in terms of the functionality with the open source. But there's a bunch of convenience features because, you know, a, we we have this persistent server in the cloud, and b, we can take care of all the infrastructure for running these sandboxes for the agent. So, like, you know, for instance, when you start up a conversation in the cloud, sandbox comes up within, like, one or two seconds rather than having to wait, like, thirty seconds or so for it to start up on your local machine. And we also can, like, connect into GitHub a little bit more seamlessly because we can have an OAuth application where you just, like, one click login and, you know, where we can access everything.

And then the the cloud feature that I love more than anything is that, if you can if you leave a comment in, like, a, like, a pull request, like, say the tests are failing, you can just say, add OpenHands. Please fix the tests. And because we have this long lived server in the cloud, that can just kick off a conversation automatically, and OpenHands will just commit back to your to your pull request. Those are actually the interactions I love the most where I don't have to go into the OpenHands UI and, like, fiddle around. I just inside of GitHub or soon inside of Slack, I just, you know, summon the agent and it just does the work for me and I get to, you know, reap the fruits at the at the end there.

What's that mind shift like and how do you get people through that?

Like, this is actually a good thing for the agent to work on or for my employee to work on. And there's also, like, a little bit of a trust thing. Right? Like, when I first started managing folks, I wanted to micromanage them. I wanted to, like, tell them exactly how to do everything, and it ended up being just more work for both of us and frustrating for them.

Once I learned to, like, trust my employees and know that, like, they might not do it exactly like like I would do it, but, like, they're gonna do a good job. They might need some coaching and some direction, but building building that trust over time is is really important, and it's the same thing with the agent. You know, the agent isn't always right. You do need to, you know, I like to say trust but verify. Right?

You need to read its code and, like, understand what it's trying to do and where it might have misunderstood something and maybe iterate a few times through either, like, a code review in GitHub or by just, like, chatting with it inside of application itself. But, yeah, very, very similar to that management experience of, like, learning to kinda take your hands off the keyboard and be really clear with somebody else about communicating these are the requirements, and here's how you can improve and things like that.

What are the technical pieces that have to be in place for you to have an agent work in a kind of larger code base or an existing project, and actually have the context that's needed to do things that fit, have the context of other things that exist in the code base, but also potentially the context of maybe it's a company style or other things like that?

But there's a bunch of technical pieces that need to go together to make that work. The underlying language model is really important. Fortunately, a lot of the core language model providers are focusing on this. We're also training language models ourselves. But the underlying language model needs to have a lot of abilities.

One kind of boring but extremely important one is the ability to edit files. So about six months ago, this was a major problem for most language models. They were not able to successfully generate a diff between what a portion of the file used to look like and what the new portion of the file would look like, or they would add an extra line or duplicate things or stuff like this. So this was a major problem. Claude is very good at this right now.

A lot of the other language models are kind of catching up to be good at doing this. Another thing that's kind of more on the especially a big problem for large code bases is identifying which files to be modifying. And this is somewhat less of a big problem than I originally thought it would be. I was imagining that this would be a really huge problem, but actually language models are pretty good. Even if you give them no tools to specifically search a code base or something like that, they use finding grep and all the other tools that a normal programmer might use and are able to navigate their way around the code base.

But I do think that code search and other things like this can help this. We have some preliminary results that demonstrate that it doesn't necessarily improve your resolution accuracy, but it definitely improves your speed of resolution. And so that's another thing. Being able to run tests and iterate on tests, being able to write appropriate tests that test whether a new piece of functionality or adding is actually working as expected or not, being able to try on the language model side, being able to try lots of different possibilities. So for example, one big failure case of a lot of language models is they try the same thing over and over and over again and get in loops and never get out of that.

Models like Claude are good at not doing this, whereas a lot of other models fall into this failure mode and don't do as well. So the list goes on. I could talk about this for much longer, but those are some of the most important parts, I think.

But yeah, maybe first, just like, why was that a step that you all felt was important and or kind of wanted to be part of your contribution to the space?

And the best closed models are somewhat expensive. There's a non trivial cost involved with using them to do agentic tasks, especially because you need to query them over and over and over again. And so having another option that's more cost effective that we can either just use as is or possibly switch over to that for easier portions of a task, but use a more expensive model for the less easy portions of the task is something that would be useful. And then customizability, we have a lot of our enterprise customers or design partners asking for some variety of customizability, be it to their code base or to a programming language that they're interested in working with and other things like this. And if we don't have a model that we can fine tune, we are limited in our scope of things that we can customize.

So looking forward, that's something that we would like to do. And we're not done yet. We just released V0.1, so we'll definitely continue being interested in this in the future.

So that's that's one really interesting path. Another path that we have where, we have a model that is specifically trained, basically to recognize whether OpenHands is on the right track to solving a problem or if it's like going off the rails. Right? So we built this model specifically based on the dataset that we've gathered. And that's a really cool product feature because on the one hand, like, you can just recognize, like, did we achieve did we did we solve the task or did we not?

And, like, report back to the user appropriately. We can stop the agent if it's, like, going off the rails, and we can say, hey. This is this is what's going wrong. Please reroute, you know, using this new strategy. We can also, like, launch several different trajectories towards solving a problem and then, you know, maybe pick one out of the out of the three that we launched and say, okay, this one looks like it's going in the best direction.

Keep following this one and kill the other two. So lots of really cool stuff we can do there by having a model that specifically knows kind of the inputs and outputs of what OpenAnts is doing.

You know, we are a company. We do wanna make money. And so we are building some closed source features specifically for, like, large teams who are using OpenHands together. But so far, we've taken the position that basically all the research we do and all the, like, know how for how the agents do as good a job as possible at solving software tasks, that should be open source. That should be available to every developer.

And it's stuff like collaboration features, things like multi tenant, things like auditing, compliance, stuff that, like, big enterprises need that your average developer working on an open source project doesn't need. That's what we're gonna hold back and say, okay. This is closed source, and we're going to enable big enterprises to do this stuff, you know, the way that big enterprises like to do things.

But then we can also take basically all the infrastructure we've built for our cloud offering and ship it into somebody else's cloud. So you can run it all inside your AWS environment. You can connect it to Bedrock. So it's basically all configured to stay within your walls.

I think you have the SWE bench and those sorts of things. But as a product, as an offering, how do you think and measure kind of that performance over time? What right now is performing very well and maybe where are those areas of improvement?

And then we're also doing things like making sure that you can fix broken commits. So you have a pull request that has failing tests and merge conflicts, and can you merge that in? And this is something developers hate to do but need to do all the time. So this is something we're putting a lot of effort into making sure we're good at, and we have some good results about that that we hope to release soon. And then other things like test generation, version updates, things like this.

The academic world is large, so it turns out there are benchmarks for almost all of these that have already been created by some institutions somewhere in the world. And so very often we talk to these institutions and say, Hey, do you want to contribute this into our evaluation harness? Often the answer is yes because they did their work for a reason. They want it to be used. So we're using that as a way to expand our vision of benchmarking to cover the actual use cases that the users are most interested in.

And you know, it's accelerating. I'd love to understand how each of you sees the future both in the space itself in terms of, you know, changing the world in terms of developer workflows, and your role in that process, as an organization and as an open source community, how you see those going forward. I'll let you guys decide who wants to go at it first, but would love to hear each of your perspectives.

And but I just I felt so powerful and so excited. And then that, like, that fades over time and, you know, it becomes a job. And then I would say over the last, you know, year or two, I've got that excitement again. I feel I feel like a wizard again. I can get so much done, you know, using large language models and using using agents.

And so I'm really excited to bring that feeling to like a whole a whole new tranche of people who have maybe had ideas for software that they want, for, you know, workflows that they want, for applications that they'd like to have and just haven't been able to like bring them to life. And I think it's really exciting that they'll they'll be able to do that. I think there's a lot of questions as to like how we enable them. Like, you know, my mom definitely has some really cool ideas, but she has no business like monitoring a production database. And so I think we're gonna need to rethink like how infrastructure works and how we ship applications and things like that.

I think there's a lot of thought that is gonna need to go into that. And I'm really excited to see kind of what what shakes out.

And I'm building agents to create software. But the reason why I'm building agents to create software is not because software is the end. It's because software is a means to an end. And I think AI has a huge possibility to increase the impact and the human condition and things like this. But I think the way it's going to do that is through software, basically.

And so if we can make it very easy to effectively create software and make it very accessible to the people who want to use it, we'll be able to make great strides forward. So that's what I'm most excited about.