Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops

Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data while keeping it all secure. Type a prompt like build me a self-service reporting tool that lets teams query customer metrics from Databricks, and they get a production ready app with the permissions and governance built in. They can self serve, and you get your time back. It's data democratization without the chaos.

Check out Retool at dataengineeringpodcast.com slash Retool today, that's r e t o o l, and see how other data teams are scaling self-service. Because let's be honest, we all need to retool how we handle data requests. Your host is Tobias Maci, and today I'm interviewing Aman Agarwal about the operational investments that are necessary to ensure you get the most out of your AI models. So, Aman, can you start by introducing yourself?

huge and all. Then I realized that that is not the way to go forward with the AI development. So I just got an idea about, let me figure out how this AI development workflow can be, like, made easy for developers. And then it started with OpenLit. And, like, this music recommendation app just got, on the back end, and then it started, with it.

So we need to be very keen on logging traces, logging most of the information, that it can help us debug the AI usage. So that's one thing. The other thing is, like, cost and token usage. Like, people are still, like, keep on using, like, AI tools, AI providers into their app, but they are still, like, very much, like, blind about how the token usage goes abruptly high, how my cost is coming, like, out to be, like, more than my revenue.

So that kind of thing is really important for us to, like, figure out. Then there is third thing is prompts. Like, people are hard coding prompts into their apps, which is, like, difficult to manage. You can have, like, hundreds of different prompts which can perform, like, thousands of use cases. But to manage that, it's really difficult. You need to, like, push in your code to the production and then update the prompts. But that is what is, like, the main area to be tackled.

work at a higher level with these LLMs, they start to adopt these various agentic patterns. And I'm wondering if you see teams who are at that level of the adoption curve falling prey to these same challenges or some of the ways that the kind of maturity curve changes the stumbling blocks that they run into.

we need to understand the AI first. Nobody is there who can understand the AI until unless you have proper monitoring, proper observability, and those kind of things to identify the areas how we need to, like, take help from the AI to generate a response to create some app. That's the main major area that we need to focus on.

and some of them might have overlap. And so then you're running into the trouble of, okay. Well, how do I figure out which part of which tool to use for this part of the problem? And I guess given that, what are some of the major components that you see as table stakes for being able to put these models into production?

it's a no vendor lock in support. Basically, any tool would be able to read that, process that, and give you output to that. So in this field of AI, like, it becomes difficult for anyone to have, like, a vendor lock in thing because people are, like, these days trying out different things, like, just before, like, launching it to production. Let's say if I am, like, white coding. I'll try out different tools of white coding until unless I just, like, get that moment,

and then I just, like, stick to one white coding tool. Similar is with the other tools, like, there is one tool, which is, like, they have this observability problem, which is sticking to their own format, which becomes difficult. Because if people are like, I want to migrate this to another platform,

they won't be able to do it. So those kind of things that people need to, like, keep in mind before opting for the products for this area, which helps them take their apps to the further level as well as maintainability is very important. If there is no maintainability, you won't be able to reach to that level where you would be like, now my product is, like, production level

and the market reaches market fit. So those are the things that people need to care about before opting for a product like this.

And I'm just wondering if you can talk to some of the ways that you think about the overall landscape and the place that you're carving out with the work that you're doing on OpenLit and sort of the comparison between what you've been focusing on and some of these other tool chains.

then they have to stick to it because they cannot move out of it. Then there is there is another one called Trace Loop, which is a very good platform as well. They are not totally open source. Their product is totally not open source. Then there is Helicon, which is open source, but they have a limitation to it with the Enterprise Edition. And what I wanted to build in was, like, to help the developers get to a particular

place where they are ready to ship their apps to a production level. It's just that there are minute things that needs to be addressed, like prompt management. And then there is secret management. For different environments, you need to have, like, different API keys for different LLM providers. Then there is evaluation thing. Then there is OpenGround, like OpenGround, which is like comparing

two different LLM providers with the same prompt. Those kind of things, if we are not allowing the users to have those things open source or free to use, they won't be able to, like, make their app to production ready. What's they'll just be left behind. So that was my, like, a notion behind building this OpenLET. So we need to have such open source product, which is, like, free to use for at least the minimum viable things so that anybody can create MVP out of it. So that was my thought.

or capture all the traces kind of thing. So we recently launched Fleet Hub, which is a feature where you can manage your multiple Oracle Collector configurations at one place. So you don't need to go into the different machine and then log in into that machine and then change the configuration, restart the server, nothing like that. You just, like, log in to OpenLit. You go and have your auto collectors listed over there, and you can just update the auto settings,

and you are done with it. When you want to, like let's say, have a, like, development app running. And then what I want to do is, like, start sending those traces to a different platform. Now you don't have to, like, go and update your app with different configuration settings and all. You just go to the hotel collector. You add your exporter to it, and it's just, like, in one go, it starts sending it the traces to a different platform. This is one. Then there is zero code observability.

We launched Kubernetes operator that is helping people to kind of without changing their app code, you'll be able to, like, instrument your app, AI app, with OpenLit Kubernetes operator. You don't need to, like, update your code, add anything to it. No no need. You just go download the Kubernetes operator through Helm, and then you can just run that Kubernetes operator. You can update your configurations, like which traces to filter out, which applications to trace

and keep the data. You're done. So these were the features that we recently worked on. We we kind of had the notion about very low user memory. Like, if you run a open lit image, you just have two components. One is click house, then the other one is, like, open lit whole client app and hotel collector and everything running. So we wanted to keep it so simple

that user do not, like, get afraid and to add those things into their application code. And they were like, I need to add a lot of things. No. Just add one line of code. It's done. So that's

and those are phrases that have a lot of nuance and gradation in terms of how they're actually implemented. I know that in OpenLET, you have experimentation in the form of being able to load two different models with the same or different prompts and do a visual comparison of them versus experimentation in terms of the AB testing style approach where you might want to route a certain percentage of traffic to one prompt or one model versus the other and then be able to track those over time.

And I'm just wondering how you're thinking about the current state of OpenLit and just some of the ways that that manual point in time experimentation factors into some of the more longitudinal experimentation that other tool chains enable.

We have the evaluations right now, but the context is missing from the open platform, which we are, like, still figuring out how to develop because context is a very, like, critical problem to solve. You cannot just provide and dump all the context to the application. It has to be, like, specific to the app. So we are just, like, working around trying to understand how we can do it. But the evaluations are there.

Right now, we have LLM as a judge. We provide like, we give it to, like, prompt, and then we give response and then understand, ask LLM to kind of give us the score of hallucination, bias, and toxicity.

But we'll just figure out and, like, enhance it. We have a future plan of, like, closing the loop. We have experimentation, then we have evaluation, and then we want to suggest what you can improve in your prompt or what you can improve in your dataset to kind of close the loop to make your AI app work, better with performance, with cost, and with, like, responses.

You can just integrate the SDK. You just send the traces to the exporters, and you are done. It's not like you we we want you to just come to our platform, use it, and then just stick to it. Sticking is one thing. We want definitely want it, but it's not like we want the users to kind of feel, like, disheartened using OpenLit. So we give out of the box features like a prompt management, secret management, evaluations, custom dashboards.

So those are the features that we want users to use and get attracted to. But it's not like if they have like, Grafana has an observability platform running into their environment. And we don't want it to, like, add in more variables to their environment. You just start sending traces to Grafana. You have the that observability platform. You just use our, like, gateway thing. Just transfer it to it.

design and development workflow as they get their initial prototypes up and running and then gain insights into how it actually behaves. How does that change the way that they think about the overall system design and the way that they structure their development versus teams who are just coming at it completely blind?

So if your tool is getting called, we name the tool. We, like, define what input was given to that tool and then what was the output to that tool, what response time it took, what tokens it took, all the information in a, like, a detailed format. If, like, anybody wants to debug a particular, like, workflow, AI workflow that is being doing, they don't need to get stuck at a place where they don't know what tool was used. At least they should be aware what tool was used. Then

they can just debug that tool if they don't like like the response. They can just go ahead and update that tool. At least that kind of detailed information they should be aware of before jumping on to creating an AI app that can be deployed to production. Because unless and until they are aware about their own product, they cannot provide services to other people.

One of the default approaches that I tend to see with teams who are building with these AI systems is they'll automatically go to whatever is the biggest and best frontier model, start building with that, get some early successes and say, okay. Great. This is what I'm going to use. And then they start to realize over time, oh, this is actually getting incredibly expensive. It would be great if I could use the cheaper model, but they're prevented from being able to do that because they don't have that confidence building machinery to be able to understand how the overall system is going to change when you go to that different model or try to optimize the prompt, especially when you're in more of an agentic workflow where you have some indeterminate

number of calls that are going to have various compounding error rates depending on the different model capabilities, etcetera.

really very important before even, like, going to MVP phase. Because unless unless until you are aware about what model to use for a particular use case, you won't be able to develop a particular solution. You will just be, like, playing around with your money and time.

have some kind of confidence into the platform because everybody is using it. If we come up with something new, then people would be like, this is new. This is unknown to me. How do I understand this? How do I do this? And the product will just, like, collapse. So that is why we built around OpenTelemetry. So there are community members. If they decide what should be changed, what keys should be changed, what format should be changed,

it's, like, common to all. And it takes a whole lot of community to kind of decide what major changes to be done. Like, recently, I remember, like, there were some kind of discussion about having some response and prompt in the event span or event events, that kind of thing. And people were, like, dicey about what to what keys to support.

And we were, like, hanging around. We shipped with different key, and then it turned out that we want to revert it. And then all those things happened. So at least the community was, like, confident enough

to come over here that at least it is it is following OpenTelemetry standard. Then there is, like, configuration thing. You can definitely add your custom configurations. You can add your custom data to it. It's not a, like, a restriction over there. We will show it. We will display it on the dashboard and everything, your custom information, but it's not, like, limiting to just the hotel format keys.

You can definitely add it. Then there is another thing which I talked about earlier, like, about the opamp management, like fleet management, managing your total collectors. Because right now, the observability is a very huge area. People are not just, like, sending traces to a particular server and then just dumping it to into the database.

That's not what is happening right now. People are having a different layer of collecting the those traces, which is eventually dumping those data into the database, like in a batch format or in a single Go format. Those configurations needs to be addressed. If we are unaware about those configurations, we would just be left behind with these total traces in the database without filtering. And once we encounter

that the filtering is not there, we would be left behind with tons of data. And it would be difficult for us to, like, filter out then. So beforehand, if it's, like, easily filterable, we should do it. That's what we have in we have in our mind, and that's what we, like, built for in the open net.

And if it errors and maybe falls back to a default prompt, then you're gonna have a high degree of variability in terms of the end user experience because you have different endpoints using different prompts to call the the API with. And I'm wondering how you've thought through some of those reliability aspects and some of the safe default behaviors around how the framework should manage the prompt, particularly in the event that whatever endpoint is serving the latest version has,

you know, high load or high latency or goes down for any reason?

that. It's, like, auto updatable, and then it deploys the app. And then on the production, when the app gets deployed, it just breaks. That's not what should happen. So that is what kind of, like, we had in our mind to support both ways. Like, if you have if you have not mentioned any version and you just want to update

your prompt to the latest, you would be able to do it. Like, you just don't mention the version, and then you just get the latest prompt. And every time you publish a new version of the prompt, it just, like, picks up that latest prompt. But similar to the Node. Js application, if you have mentioned a particular version, then unless and until you update your app with a new version, you won't be able to access that prompt. So that kind of reliability is very important when you are developing an AI application. If you just, like, update a little bit of context into that prompt, the whole meaning of how the LLM would understand

it, it'll change. Because LLM, it's not like you would be able to guess what LLM actually refer to a context. If you have thousand lines of code, you provide a LLM thousand line of code and you ask a particular single particular thing, you don't know how that

LLM gave the answer, whether or not it referred all the points mentioning that particular question or it just picked up one line and then gave you the answer. You are not sure about it, how the LLM would work. So it's very important to, like, frame your prompts in such a manner that it doesn't lose your context, and it's reliable when any update happens. So that's what, like, the prompt management is all about.

And then the journey began with the OpenLit as a whole new product, an AI engineering tool rather than being just an observability layer. So the scoping kept on changing. Earlier, we didn't support OTL format. Like, two years back, maybe when OTL launched, we just were the one of the first ones to support the OTL format. We are totally open source right now. We don't have any enterprise version, any hidden cost, anything

like the other players in the market. Then there is, like, lot 50 plus of integrations that has happened, and it it will kept like, keep on increasing. Then there is, like, the open ground. If you, like, look the previous version of the OpenGround, that was very crude. We were we just, like, wanted to, like, have that feature. We built it and we launched it. Now if you see OpenGround, it has a lot of, like, integrations between the LLM providers. Then there is

this fleet management that came up. So we have a couple of things on our mind, which is, like, having datasets, having the whole new evaluation life cycle of, like, understanding, adding context to it, all those things. And then there is a couple of other things. Right now, we are totally focused on, like, pushing it to open source. We are not like, we will keep this as an enterprise thing because once your product is, like, there to hit the market, what I feel is,

like, if you're happy with your product, it's okay to, like, give it free. Eventually, you'll get to a point where you can earn, like, out of it. You can monetize it. There is no line where it's like now you cannot mon monetize it. Anytime that would come like that place would come where you are not like, now this is the feature that I want to, like, charge user for, but that's okay. Like, if if I charge less, that's okay. So that's that's the, like, thought process over here.

It's an overall tool which helps you build AI apps faster and more reliable. And the other aspect which we are thinking of is which every other company is, like, thinking of, which is data security. We don't want to, like, use users' data. So that is why we don't have, like, exact numbers, how many people are using it and how what is the exact count of events. We just have a rough estimation. We just mention it on the docs page that we track

some data, but not the user data. We don't even track email, user email, through which we can just understand what the user which company the user is working on. But we want to stick to data security as a huge, like, aspect, which is a, like, a trust building factor for us.

afraid of, like, just dangling all these traces data into each other. It will be separate. It will send separate date data, and you'll be able to, like, visualize it properly. You'll be able to evaluate each individual traces so that it's easier for you to understand what way my product is going towards,

which I can improve on. If my product is, like, deviating from the path it should have, then I can just, like, modify a couple of things using open telemetry data, those toxicity evaluations, bias evaluations, and then there is analyzing all the individual traces of each step for a particular LLM call. Those kind of detailed information would actually help the users to kind of polish their AI apps.

I provided it all the information, but it's failing. How and what to tackle? Those are the kind of blind spots that we think that it needs to be addressed soon because until unless you tackle the context issues, you won't be able to get a meaningful response out of it. You will just be fighting over the different same prompt, different responses, and you'll just be, like, wondering, on the top of the mountain that should I jump or not, so that kind of thing.

into their own platform. They added that database configuration into it because we have a DB change and DB added event kind of thing. So we knew, like, lot of DB configurations getting added. It was surprising to see that it grew. Like people have been like pushing in database configurations like in in a huge number. So that was the surprising edge for me. Then there was like how people are, like, using OpenLit.

They were trying to send data to different platforms via OpenLit, and they were trying to, like, send it to different exporters. And people have been, like, doing it for lot of other platforms. That gave us an idea about to go forward with the company and then just reach out to the different companies and ask them about the integration thing. Recently, the Dash zero also added open late integrations.

Last year, I remember, like, I had a conversation with the one of the Elastic guy, and we, like, wrote a blog about how we can, like, observe AI apps using Elastic and then open it, that kind of thing. And it it was a good response from users reading out that blog. So it's very, like, surprising to see. People have been mentioning OpenLit as a, like, a requirement for their jobs. That was, like, a very good moment for me, like, because this thing, OpenLit, has now become a,

like, a requirement for a AI developer kind of thing. So that's that.

with feature request. Like, I want this provider. I want that provider. I was like, how do I, like, support all these information to it? So that was difficult for me, which helped me understand that catering all the requirements, all the features at once is not the part of the plan. You need to focus on the experience first before, like, addressing those feature request.

We shipped OpenGround very crude, and it wasn't attached to the other components in the dashboard, which actually, like, we were trying to, like, address that very frequently, but we couldn't. And then we tried modifying it, but that had the prior less priority.

So it took me around an year to address that and then come up with version two of open open graph, which was very disheartening because people might not be using it. So why ship? Ship what has a very good experience, developer experience. Otherwise, people would just be coming and going. So that is a best learning I could get, one of the best learning that I could get from it.

supports the generic use cases, open telemetry format, which is, like, getting adopted by every other person. But your app might not be able to fit in with those requirements. So first of all, that and then there is, like, if you want cloud support, we don't have cloud support right now. Eventually, would, but right now, it's not there. Then there is couple of things like gateways. We don't have gateways. If you want gateways, like API gateway kind of thing, you go forward with other platforms.

But all in all, what we have is, like, worth trying for. You can try. You can decide. People can just tag me with what what are the feature requests, and I'll think about it.

So that is the very keen, like, area that I am currently focusing on how to, like, tackle that.

The resource probably, like, remains for for a lifetime. So that was the use case that we were trying for. But other than that, I think I mostly covered what I really felt.

the AI just just rules the world. So we need to be more emotional, and we need to increase more emotional connection with the human rather than just trusting AI for, like, giving us the response. And

the other thing is, like, the memory. We need to provide some memory to the AI to be able to, like, get things from the past. But we, as a human, we have that memory in built in our system, which is like different type of memory, short lived, long lived, and all those things. And that we need to, like, respect before, like, like, pushing all those things to the AI because, like, AI could, like, ruin what we, like, enjoyed in the previous life. So that's that.

with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.