Monologue: Everyone Suddenly Cares About AI ROI

that mostly comes down to one thing. Hey, does anyone have any way to measure the ROI of AI? Didn't fucking think of that one digit fellas, Oh no, I want to come up with that before I spent all the fucking money. But I don't know. I don't run a fucking business, drap. That's because up until recently, most companies have been able to pay for a subscription with subsidized token rates, meaning that forever dollar of their subscription they could burn anywhere from three to thirteen dollars worth

of tokens. They just had rate limits. Whatever they did just eight into those rate limits, and they didn't really think about the token cost in fact, every one of

these companies deliberately obfuscates that information. You can find it through CC usage, which you need to have command lined interface, and there's also slash usage with anthropic Regardless, as a result of all of this, nobody was really measuring the actual positive outcomes of AI services, or even how much a particular task would burning tokens, which means that with the advent of token based billing for enterprises, everyone is just guessing about how much their annual budget should be

based on virtually no useful information. Imagine you're doing something for years, years and years, and you're claiming it it's the future, and it's going to change everything. It's going to change your business, and there's other businesses. It's the future of your revenue and their revenue. Oh my god, it's so amazing. But not once did your clever nbaar say,

why don't we make sure it actually is? No one did that, not one of them, Not one of these fuckers have been I've been checking a week, we're talking to people a week. In truth, though, measuring ROI is pretty difficult for a lot of knowledge work because it requires you to actually know what a person does for a living, and what doing a job is like, and what the outcomes have said job are and what the

actual like good results at the end may be. Which is a lot to ask for middle managers and CEOs who mostly go to and from lunch and explain why other people's work is actually theirs. And I want to say, if you're a middle manager hearing this, more than likely I'm completely correct about you. Most middle managers are fucking scum. They exist only to steal work from people and nag them and harass them. If you're not one of those, great you shouldn't be offended by this. This is not

about you. But if you are, you're probably one of the people who's going to email me, in which case, wei, weare God, cry to someone else. I don't give a ship, but the lack of measurement of both task costs and

ROI as executives already crying for mercy. An engineer at a popular payment processor recently TOLDY that their organization has in the last week burned over one and a half million dollars in anthropic tokens, with one user, one person burning over one hundred thousand dollars in that time period.

Over at Zilo engineers are both through ninety five percent of their annual cursor budget in less than five months, much like Uber's CTO revealed to Laura brand Over at the Information they had burned its entire annual token budget by the middle of April. Axios's Manny Mills reported that a consultant advising a CFO at an unnamed company said that an organization had spent five hundred million dollars in the space of a month on anthropics tokens. Jesus fucking Christ,

Jesus Christ, Jesus Christ. I'm sorry, I know I'm ranting. I guess that's what I do on these monologues. But you spent half a billion dollars. You fuck nuts. Have no idea what you did that, did you? None of you'd don't. None of these companies do. They don't. They don't have any idea why they're doing this. Executives only do this when they're very sick. We got we've got to give them some cap milk or maybe take them to the vet to finally deal with the problem ourselves.

But the AI industry doesn't really have a response to all of this, other than to say that somebody could theoretically make a way to measure ROI in the future based on some metrics that nobody can can explain. I've read thousands of words now of random Twitter posts, mostly AI, generated of course by people that don't realize that it's obvious say that these huge posts, and they always like, yeah,

you know, the next thing is in AI. It's going to be tokenomics measurement and extrapolation of ROI measurements from there. They're just saying, yeah, at some point, maybe we need to know why we're doing this, not just the because we all love AI. It's like the end of it, so the beginning of Death of Stalin, when they're all standing around being like he looks great. Everyone's just like, yeah,

of course we all love AI. Of course we all know this, and we all know that AI is the most special and beautiful thing and it's changing my life, of course my life as well. Yes, it's so amazing, but like, do we have a way of measuring if it's actually doing that? And no one does, not a

single one of them, does. You See the problem is they don't want to face that everybody should have measured ROI from the beginning and only avoided doing so because subsidized subscriptions allowed everybody to ignore the problem right up

until it was too late. Anthropics supposed explosion of growth appears to have come entirely from experimental revenue organizations that have no way of measuring costs or return on investment, run by business idiots who don't know what real work is, directly incentivizing workers to use more AI right up until their CFO says, hey, I just saw we spent tens of millions of dollars on this. Is it producing anything of value? Like? Can you can? You just can? One

of you? And what's insane? Is I would expect And never in my wildest dreams did I think that it would be everyone. I thought that there'd be a couple of organizations who could be like, they have a messy metric, they've hammered out, they've really they've bone smashed it like clavicular but with metrics, or we're going to make this look good and then we'll be able to say this number went up. I thought they'd have a few. I thought they'd have some sort of weird metric that they

came up with. They don't have to be talking to the fuckers all week. I've been asking them, and no one does. In fact, at one company they were literally saying, Yeah, the way we measure someone's AI use and whether it's good or not is how many pull requests they've done via AI, and Paul Cress being someone's gonna email and be mad about this, calmed down. The thing that you do when you say, hey, here is the change I want to make to this code or this thing. This

is the plan. Do you agree there? They're measuring the percentage of those written by AI. That is a positive metric. That's what they're measuring. I didn't think no one would have any idea, though, I truly didn't. It's so funny, it's so fuzzy, it's so fucking funny. It's the funniest thing. I'm right. I've been right about this four years. I've been saying four years that when organizations pay the actual cost of AI, not subsidized subscriptions that allow you to

bone effectively as much as you want. They called the start asking questions. All of these people who've been talking about how amazing AI is are now going, well, you know, we need to have very difficult or we need to we need to measure it somehow, and the value of AI has been entirely framed around unsustainable, unprofitable, impossible to measure, tools that beguile, imbeciles, and those that don't want to

think about reality. To be clear, open ai also appears to be moving most businesses towards token based billing by giving them one thousand dollars in credits to use on codex. To soften the blow, fucking Sam, No, nobody does it better. Nobody does it better than Clammy Samultman. Clammy said glamy Sammy. We love Clammy. One thousand dollars just for you, honey. You can use it whatever you want, Please pay me. And that's the thing. Open ai broken business as well.

But there's no real solution here. There's none, And I think the next three months are going to be illuminating and probably feature pullback across a ton of organizations as they ask questions they should have been asking since fucking

twenty twenty three. It's insane, I would as I sound completely insane myself, as I sound completely crazed, but people will calling me crazy in twenty twenty four, being like, yeah, you know, when businesses pay the real cost, They're not gonna People were acting like I was a fucking lunatic. Well who's crazy? Now? Would a crazy person laugh like this? Ah? Anyway, there are no real solutions here though, folks. There are

no real solutions. While somebody theoretically could move their workloads to deep Seek or a cheaper model, it's unlikely that there are inference providers that can support that much traffic or even provide that much stability. You have to remember, a big organization is putting a lot more pressure on deep Seek than say a casual code of someone who's just running their own GPU, and you have to do

far more than just turn on GPUs. You actually have to build for an organization that would be theoretically spending millions a massive inference stack that cannot crap out. It's one of the reasons people pay Anthropic and open AI, even though Anthropic is barely stable. I also don't have any substantive proof that deep Seek V four or other models can replace an pa opus I say, opeth there not editing that or Codex model, even if they are

dramatically cheaper. We have no proof of this, and despite the conversation, no one's doing it. Not heard it, not heard anyone say this. So in the end, if nobody can measure the ROI of AI in general, why would they spend anything now. AI boosters are currently either completely ignoring this subject, which is what's happening with the obvious ones, or they're going to answer by saying that we're in the early days and that there are weekly breakthroughs that

will solve these problems. Be in a lot of fucking weeks so far, been a lot of them, what like a four hundred of them, four hundred and fifty coming up with five hundred weeks. But this is a problem that needed to be solved yesterday, and neither anthropic nor open AI has an answer for bringing down costs, let alone those of their customers. Even if even if they were magically profitable overnight, even if they were, they're not.

They're not profitable in inference, their customers are spending too much money. Their customers do not have cost control. That makes it very hard to budget for how much you'll spend basic business stuff, basic stuff. I'm not innovative in my thinking here, you know. Is it's like that tweet that was like, yeah, walking around a college campus looking into a business school class, and it's like profit is

revenue minus costs. It really is that, except these people have been ignoring that four years and making people feel stupid for asking those questions. I also must be clear that these companies cannot slow down. Anthropic and open AI project to have over three hundred and fifty billion dollars in combine down your revenue by twenty thirty, and they're gonna need it, because between them they've made over one

point one trillion dollars in compute commitments. If their revenue growth is entirely based on dimwitted executives allowing a no it loads refused compute dump for no apparent return on investment, that is not a stable, sustainable, or even a growth business. It's a grift targeting a society wide executive ignorance of production itself. I look forward to telling you more about it next week. I've had a lot of fun this week, got Paul Kedrowski coming back on the show on Tuesday,

actually Wednesdays, sorry, twelve am met Wednesday. I should really know that by now it's been years. And then yeah, I'll have another monologue for you. It's been a lot of fun. I hope you're enjoying it. ZiT run Out