Go wild. So that's how the episode starts with Jillian Wild. Speaking of which, staying on the internet, Yeah too late, it's too late. Yeah, okay, Speaking of which, you are back on the show with us. You're back in the US, So it's been going on.
Well, I mean I moved continent, so that was that was a pretty big thing.
Now I'm in the US full time, which.
Is exciting, Like just two hours north of Boston and a little town that probably nobody has ever heard of in New Hampshire. But it's been fun.
It's been it's been a big adjustment, but it's been a really good time.
Yeah. From you were in Doka.
Right Doha?
Oh yeah, yeah, in the Middle East, So yeah, Doha to New Hampshire. There's some adjustment there.
There, there is. There's a lot more color here, like just in general the story.
I'm sure. Yeah, I've not been to Dolla, but I have been to Dubai several times, and I think they're fairly similar.
Yeah. Yeah, it's like less than an hour's plane ride. They're about the same thing.
Yeah. Cool. So back in the US permanently or is this a temporary thing for you?
Permanently? No living here now?
Right on. Well cool, Well, welcome back home.
Thank you. It's been nice.
I'm very happy to be back, and it's nice that professionally, it's so much better for me too. I don't have to work split shifts and that's great, and I get to be back on the podcast and talk to you guys, which is obviously.
The biggest win.
Of course.
Yeah, of course, go harass my clients in Boston. Now that's where most of them are at and here we are.
Oh but two hours like and you know, so you said maybe no one knows about that, but I actually made that drive fair number of times up to Lake Ossipe, So there's there's a whole bunch of town there that I could probably rattle off. And maybe maybe you're living in one of those. That's a long I get from New Hampshire downtown.
I'm not driving, though, so it doesn't bother me. I drive to the bus station and then one of those very nice buses takes me into Boston. I'm not trying to drive in Boston, Like, that's not a thing that we're doing.
It doesn't really matter, right because you're on the East coast where they actually have public transit.
No, the bus is actually nice, Like I don't actually mind taking the bus. It has like the really nice big seats and it goes right into South Station and then I don't know, most of my clients are like ten fifteen minutes from there.
That's cool. I do kind of miss that. I miss that every once in a while of just seeing and working with people face to face.
I do. I actually do like that, Like I wouldn't even really want to have a hybrid job, but showing up like I don't know quarterly and just going to go see everybody. Usually it's for some type of meetup. There's like, you know, a big meeting of the mind's happening or a conference or a talk or something like that, and then I'll go for sure.
So what was it like moving across continents? Did you like bring like couches and mattresses and furniture and did no?
No, no, it's all I don't know.
I guess like living overseas, all the furniture and stuff got treated as kind of disposable because there was enough of those moves. And the last place that we were at was the last place that we were at was mostly furnished anyway, So I didn't have a lot of furniture. So I bought like furniture when I bought the house that I'm living in now right on.
Yeah, cool, We did that whenever we moved to Oregon from Arizona. The people that bought our house in Arizona wanted to buy most of the furniture too, So when we got up here, we had almost no furniture. And so the first thing we had to do was, you know, go drop thousands of dollars on furniture. And I can't say we made the best decisions, you know, because like you're like desperate for a couch and a mass so you don't really research it or take time to say
do I really like this? You just like say I think I'll like this, and then three months later you're like, I probably should have gone a different direction.
You're an amateur mover.
That's fun.
Yeah, I mean I do, because I'm insane.
Like I would just buy some of the Amazon like Futon mattresses, like the kind that I take when I go camping, and I would throw those on the floor with some pillows, and that's what we would have until I found a couch that I obsessed research and I have like a whole decision matrix of couch versus size in the living room versus how I'm gonna like this versus different seasons, and how I'm gonna like it? How much can the destroy it? Like, there's a lot.
That goes into these decisions for.
Me, So you figured it out.
That's not how I do anything, ever.
The futon sofa and I also used to buy a queen size blow up mattress because you know that's fairly cheap and can help you a lot there with even knowing what your situation is going to be. If you can't find a quick futon sofa and then yeah, then you've got as much time as you're willing to sacrifice into living into this terrible situation until you actually define the mattress you're looking for, the furniture you're willing to I got a lot of black previously because I used
to use my moving boxes as partial furniture. This box turned sideways is like a shelf. You stack some boxes on top, and then you have a bureau made out of cardboard.
There, he does.
Although I am willing to fight you on the air mattresses because air mattresses are like one hundred bucks now, and my kids break them fairly regularly, so they don't last long enough for us, which is why I have moved on, which is why I moved on to the futon mattress like lifestyle, and I'm willing to commit to that.
Yeah, for life.
That's it.
Nothing bad is going to happen to host mattresses ever, right on.
Yeah, cool, Well, welcome back. Excited to have you back on the show.
And thank you for letting me back.
Yeah, of course, anytime, especially for today's episode because we're talking about the twenty twenty four Door Report. Because I mean, by the time this episode releases, I'm sure everyone will have read all one hundred and twenty pages and this is going to be a pretty boring episode because you
will have digested all the information already. But for the two or three people who didn't read it, we're here to get you at the speed so that you can converse with everyone else as if you spent just as much time reading it as they did.
So I think instead of people reading it, we're all going to have to brains and then the report will automatically be wrong because it says there ai ho's and how as much have been impact as was expected, except in Drug Discovery, where it's been insane. But what could happen?
I honestly, while I was reading it, I had this fear like, oh, you like, our whole episode could have just been feeding the report into notebook LM and having that running instead of the legitimate adventures and DevOps for our you know, faithful viewers here.
Well, I haven't actually read, so.
I did go through. I did go through almost all of it.
There were some pages at the end that I skipped that was just repeating a little bit about what Dora does and for those that actually don't know, Dora was a separate company that Google bought in twenty eighteen that was focused on the delivery metrics that we all know and now love in and around engineering to have.
I mean, there's nothing else.
I say, there's nothing else, although there are a bunch of companies that have started to experiment with a sort of revision of them called Space, which I don't have a lot of experience with, but does handle additional aspects that some of the Dora metrics are missing.
Is space another acronym for something?
Yeah, you know, because people love acronyms and they don't mean anything, and I'm sure it stands for something, but I think realistically most organizations can do a better job by just understanding where their metrics are currently at and focusing on improving them irrelevant of what they are, because
no metric is entirely perfect. Because of the companies that I've worked with in my history, often what happens is they say, oh, we're we have great Dora metrics, or we even have bad door metrics, and we need to do better improving them. And then I look at how they're capturing those metrics and they couldn't be more wrong. Like you look at them and they're say, let's say, using feature flags and whatnot, and so you're looking at
your time to roll out changes in production. But those changes in production have never been tested, they're never being used, and they're behind flags that aren't even being that don't make that functionality available to your customers. And so it's like, can you really even say it's in the production. And yet these companies are saying, oh, yeah, we roll out our code immediately. It's all in craud, it's all being used,
and so our values for that are really high. So I mean, I think in those cases that it doesn't matter what the metric is if you're recording it in a way which doesn't encourage good engineering practices.
So I think maybe we should backtrack that a little bit. What is a Dora metric?
Dora is this little girl in a cartoon with a monkey that gets chased by a fox, and evidently she does reporting in between episodes of making cartoons.
That's fair. I'll bet she'd be a better reporter than most.
Yeah, that was a reference to Dora the Explorer. For anyone who didn't get that joke, or.
Maybe before my time, it was gone.
Yeah, it was when my kids were young, so it's probably pretty old at this point.
My oldest really went through like a phase where she really loved Dora.
She was like three or something.
That was very cute. It was one of the shows that I actually liked too. Not all of them, but some of them are adorable.
I'm sorry, I totally derailed the topic. What was your question, Jillian?
Yeah, what is it? Door? We're talking about Dora metrics and it's like, well, what is a Dora metric? First of all, I mean, I suppose we can all kind of infer it, right, It's some type of metric that we're using to and for how devopsy we really all are, Like who's got the highest score? Is there a scoreboard that we can we can enter ourselves into, Like what are we doing here?
So there was actually like four official ones that everyone could could be using. And this is where the test really comes because I don't know if I remember all of them off.
On my head.
But there is a mean time to resolution when a failure happens and you need to go and resolve it and get your production state back to working according to your SLAS. Then there's cycle time to deploy changes from the time you start working on them. There's change failure rate, how often failures actually happen because of the changes you're making. And I missed one and I'm hoping Will knows which one I missed.
Yeah, we have change lead time, deployment frequency with frequency yeah, then the change fail rate and deployment recovery time.
Yeah.
Yeah. So it's interesting. It's one hundred and twenty pages of texts that just boils down to those four metrics.
So normally, normally that's what they talk about.
And so the question is that since they've been doing this what's new every year that is relevant because once you sort of share that these are the metrics that every engineering team could be utilizing to decide like how great they are. And they talk about benchmarks and where elite teams are versus maybe behind the curve teams that have a long way to go, what changes from year to year. And so this year they dove a lot
more into three areas. The first one was they started talking about a new metric the amount of rework you have, so when you thought something was completed and you have to go back. And this is really interesting because it was always something that made a lot of sense to me and wasn't included something else that they don't really
talk about. It's like the number of support tickets you get, like whether or not, Like there may not be a production incident in any way, but your customers may be concerned and confused, your users.
May not know what's going on, and that's not really included.
So they talk about expanding the metrics and maybe breaking them down into different areas that are all independent of each other. And that's sort of what's important, right, Metrics that depend on each other or have high correlation or high cohesion tend not to be as effective because they measure.
The same thing.
Then the report moves on to in depth review of the impact AI in our industry and the world, really about engineering teams and software, and the last part talks a lot about platform engineering and its impact.
Yeah, before we dig into those, I think the thing that stood out to me most in here is the section applying insights from Dora, and it says the recommended approach is to identify the outcome you would like to improve, measure your baseline or current state, and then I feel like this is where so many places start falling down.
So the next step is to develop a hypothesis about what might get you to your closer state, and then agree to a plan for improvement, do the work, and then the last one measure the progress that you've made. I feel like that's where we fail a lot of times. You know, we like say, hey, we're going to make this better, and then we start working on it and never take a step back to say did we actually
make it better? And if not, let's rip that out, And so over time we end up dragging all of these failed attempts with us adding you know, layers of complexity or bureaucracy or whatever the addition was, and we just keep dragging it along with us, even though it doesn't work for us or improve the situation for us.
Yeah.
No, I mean I think we bring that up a lot in this podcast, and I think the report really dives into it in the executive summary section. Which is an experiment is a hypothesis that is time bound with retrospective at some point. And often I think we do see a lot of organizations say, oh, we could try this thing, which is great, like you should absolutely experiment, but then there is no review, like was it even a good thing.
For us to do?
And that's like there is no like any experiment is a success, even if it rejects your hypothesis, if you come out with some learning. The only failed experiment is when you start it and never stop. And I think that's what a lot of organizations end up doing. They try something out, they experiment with a different team structure,
even something that we believe is good. You know. I think it was a few years ago that the concept of team toypapologies came out with different types of teams and how they will work, stream aligned teams, platform teams, complex subsystem teams, etc. And it's a great idea to shift an organizational structure to focus on these, But even if it's a thing that everyone agrees on, you should still be like, we believe that a different org structure would make sense for us, and then measure why that
is the case at some point later, like did it actually have an impact? Because if you don't do that, someone else is just going to say later, oh, that was bad for us, and you're not going to have any data to really back that up. A common argument against that has been the notion like rapid reteaming or
dynamic reteaming. I think it's a book on the topic, actually, which is this concept ofok, like you swarm on an idea, you just get it to completion, and the you, you know, di spand the team and create a new team to focus on it, and without really understanding when your core businesses or why you're making the change, just arbitrarily making changes your organization, you don't know if it's going to be successful in the end.
So my favorite way of thinking about applying the scientific method is when it involves people, because the only way that you can actually do that is to spin off parallel universes and have this whole double lunch where you have all these simulations of different people, and then you have your crystal ball where you're like, well, how did
this actually work and how many times? And then you have the different iterations, and then you would have different like branching models of the different iterations too, and since you can't actually do that, you just get to guess. But at least with technology, we could have isolated environments where we really applied the scientific method. But you can't so much with people because people are chaos.
I mean, you're absolutely right, Jillian.
I feel like most companies don't have the sort of scale to be able to do this, and it's a mistake to believe, especially if you're like selling to businesses, that you have the ability to segregate a clear control group and a test and multiple test groups to validate
what's going on. I do remember there was a not great book that referenced one of the major tech companies in the world today, and they had talked extensively about horse trading, individual countries, populations, as far as users go to perform human psychological experiments on the information that the company was providing them.
To see how humans would deal with that sort of.
Information, and so like at scale, you know, there is something, but that's not like most of the companies, Like it's not my company. I don't think it's well as company. I'm sure it's not yours, Agilian, Like most people don't work at a company where this is actually a real thing that could be done at a human level scale.
No, I'm also uncomfortable with idea of like the human experimentation, Like my scenario was a lot more fun because I would just like to point that out there.
So what you're saying is it's okay to experiment with humans as long as they're in parallel universes, just not in this one.
Yeah, fine, I mean we agreed, that's all fine, Like we all like the crossover events and the Arrowverse, like this is what we do.
Okay, I wouldn't.
I wouldn't have mentioned the arrow Verse, but you know, I uh yeah, I mean I I'll just have done quite heavily in science fiction and fantasy, for sure. I Mean I always liked the question like did you did you enjoy.
Being an older sibling?
Because I only have younger siblings and I'm like I honestly don't know how to answer that question. I've never been a younger sibling, so I have.
To see this is what I mean. We can't spin you off like we can't. We can't create like another experiment with you as a person where then you have the perception of having older siblings, but it doesn't work. I could spend it, Listen, I have a neuroscience to could spend like a lot of time mark doing about this, but we may.
Want the actual devox portion of the show.
I mean, I'm just wondering if there's, like you're secretly developing some sort of medical treatment here that like would cause you to forget your memories and still be able to persist in a new hypothetical reality where you could actually.
My second time, oh that exists, Go smoke some weed, wake up in a room with a bunch of empty cereal bowls and boxes of fruity pebbles, going Wait, right.
Is memory loss associated with potsmoking?
I've heard that it is. I don't know that for a fact.
I mean, maybe maybe we're missing out on this prime experimental tool that we could be using in legitimate professional.
Applications, absolutely legitimate, I'm sure we could get lots of sign up for that and it'll be fine.
Well, ethical dilemmas there.
Well, what they did in my university is they used to run these experiments all the time.
And all they do is they say that.
You know, well, you have students, the students like care about that.
And you pay them, Yeah, students that you pay like twenty dollars per study, and they will for sure sign up, sign away all of their rights and do whatever unethical thing you want in your treatment. No co.
Yeah, that's how that goes.
Wow, I'm glad we don't get real time analytics of our episodes. I don't know. Maybe maybe this would be a maybe this would actually increase our ratings. I don't know. Meanwhile, don't know.
But the whole like organizational team building and all that stuff, all of those are very interesting to me because like, you can't really test it properly, and the only proper way to test anything is to use the scientific method, so you know. But then it's like, all right, so you can't test it properly, but what can you do
to kind of approximate it? Like you know, obviously this is not black and white there, they're like ways you can you know, evaluate the performance of you know, of team organizations or any of that kind of stuff.
And for the purposes of.
This talk, we should probably you know, just just stick with reality of like, Okay, these are the things that people are doing and what are they doing. So was there anything in this report that discusses like, well, this is how your team should be organized if you have you have a platform team and an AI team and a.
So it definitely tries to people the team.
So I think this is one of those things where it's research oriented, not so much prescriptive or evaluating what happened. So the best it comes out with is this is what some people are doing, and this is the data, and here's us trying to maybe explain why the data is what it is. So there is an aspect of a belief that a lot of organizations are creating something that they're calling platform engineering or platform engineering teams.
And obviously some of those organizations just took.
Their age old infra team, didn't change anything or the people or the mindset and just rebrand, you know, slapped a new label on it and said this is a platform engineering team.
We're goal then. So I mean, obviously.
Those people and what they're executing on and their approach. It didn't change, and so they're likely going to be in the laggard bucket.
And low performers.
And then there's other teams that are other companies that really focused on how do we elevate what we're doing, how do we And one of the things they really care about is if we look back at the door metrics, is maybe something called developer productivity and what does that even mean realistically? But maybe it's the number of lines of code that you push out. They don't really talk about this, but it's how people feel. So maybe it's the number four requests or the number of GERA tickets
that got completed. And no one uses Gira obviously anymore, so maybe they're using like linear, so linear tickets that got completed, and we can evaluate number of tickets that got completed equals velocity thereby productivity, and so did our organizational change that we made cause an increased number of tickets that get completed? And when it comes to platform engineering, the sort of conclusion is that it doesn't have as big of impact as we wanted to or believe it should.
And even in the.
Best companies it about averages out to if you have some sort of platform engineering team and they're doing a great job, it amounts to about five percent productivity increase
per engineer. So that's five percent across your organization basically, and I think this is a really interesting number if you think a lot about it, because it means that if you have twenty engineers, you can hire one person to power your platform engineering team, because if you hire more than one person, then you're not going to be able to increase the productivity of those engineers any more
than that. The five percent times twenty is one hundred percent of the amount of work that could possibly be done. And of course you don't want to have a team of one person. You probably have a team of four people. So realistically, if we just look at the numbers, having a platform engineering team before you have one hundred person organization means that you are spending too much time focusing on whatever the platform engineering.
Team is doing and optimizing for things that the.
Actual who your users are, the engineers, developers, product engineers don't get the benefit out of it. You can't increase the productctivity.
Further, Yeah, I would agree with that because I think in those like the scenario were describing. You know, before you have one hundred engineers. I think for most companies in my experience, before that, you don't even really know what you're doing as a company. So it's it's hard to optimize for something when you don't know what that something is.
Yeah, for sure, Julian, you're gonna say something.
Oh, I was just going to say, you were talking about the difference between platform engineering teams, which is kind of a I would say, maybe like a newer phrase, and infrastructure teams, and I was just wondering, what do you what do you kind of see is the difference between the two of those.
I mean, this is this is where like you know, it's because well, I mean no, I mean, I think it's a really great question because at least from my engineering experience, and I feel like I'm a little bit unique here because when I started, some of the things I was doing, I would definitely say came into effect because of the mindset of DevOps was growing up now getting out of it seed stages, and the work I was doing was specifically to eliminate teams in the organization
that we're called platform engineering teams, and now almost twenty years later, we're now creating platform engineering teams and calling the platform engineers because I think everyone no one remembered that this term had been used previously, and they're doing similar things.
But we've automated a lot of.
Whey so much to the point of where we feel like the job now could exist again, but doing different things. So when I say infrastructure, what I had meant was is maybe writing some scripts to automate the installation of some software on bare metal racks for our servers that you got installed. Often they may even be responsible for setting installing the blades and putting together the server blades in the data center, but for sure writing the scripts
to manage that and maybe send alerts and whatnot. And today I doubt any platform engineering team is doing that work. I mean, even when they have an on prem solution and an on prem data center, they're likely not even
building the blades. You're outsourcing data center creation. Even if you have a physical onpreme data center, you are relying on these third party companies that come in build it for you, of which then you're likely not even installing the base software, but then you're installing some sort of native system on top of that, either at Kubernetes.
Or Coros or something like that.
I think I think the end product for both of those teams is the same, you know, to create a place for the engineering teams you support to run their applications. I think the big difference between the two is how they accomplish that goal or as an infrastructure engineer will be more hands on and more tightly interfaced with the
infrastructure components themselves versus a platform engineering team. It abstracts themselves away from those those complexities and will rely on like either one or more platform engineering style tools that automate or abstract those tasks away from them. So I think I think the end product is the same as the approach of how you build that end product distinguishes the two teams.
Yeah, for sure.
I mean I think my CEO and I were talking about this recently and she brought up there's like a lot of different other things here that probably aren't necessarily normally associated with it, but things like building golden paths of sets of frameworks and tech stacks that we decide to use, but you know, someone has to do the difficult work of actually configuring it. Because these things are too unopinionated about what you're utilizing is one of them.
Maybe you're utilizing some sort of shared digital or software driven infrastructure, infrastructure as code and coming up with those common components that go together would be another one. Sometimes there's an integration with IT system so SSO based integration from your IT provider, whatever it is in Google Workspace, et cetera, to power as SO in your organization, and the integration between other apps you have, or the management of the cloud accounts if you're using AWS or another
cloud provider. And obviously those things didn't exist twenty years ago.
So I'm not opposed to using like the newest buzzword, mostly because I do that every couple of years on my LinkedIn to get myself a raised Like that's all fine with me, But I specifically don't like the word
platform because it's just it's too general. And then I'm in you know, these meetings where you have like the scientists, and then you have like the infrastructure people, and then you have like the drug discovery, and you have all these different groups and they all use the word platform and it means something completely different to each group, and then everybody's just having their own conversations, and they don't seem to realize that there's like four different conversations going
on because we're all using the same word, but it does not mean the same thing. That's my rant about platform, which I suppose doesn't maybe have a whole lot to do with the Dora report, but I don't like it.
It's too general a word.
I think.
I think you're you're totally on a good point there.
Actually, So last year I ran a seminar at one of the DevOps Days conferences trying to define actually what a platform is, and after an hour talking on the subject, the best we were able to come up with is it's something horizontal that supports something else, which.
Is just.
The beams of my house.
That's it.
Isn't that the foundation though?
So I mean, I don't know if the analogy broke down what you're talking about.
Well, we we got platform.
I think the best analogy we had is if you think about the gates for getting on a train, like the plat those are the platforms. You stand on the platform to get to the height of the entrance to the train. So the train if you had to board it from the ground layer or floor you know whatever from the earth. Uh, it's too far away. It's not well designed from that regard, and so the platform lifts you up to be able to utilize that technology as
efficiently as possible. And what it doesn't handle is you think about that as a platform, it's very difficult to raise up that concrete slab or lower it. If the height of the train changes, or if the types of the trains change, or if the train is in a different location, you may have to extend the platform.
And so this is I think Will sort of brought this up.
One of the things that's actually highlighting the report is that platform engineering teams, what they build at first, tackles a lot of the low hanging fruit, and that's things that are easy, that have a lot of high return on investment that the engineering teams may be struggling with. Right, you see the same problem a bunch of times, and then you develop a solution to help solve them. Maybe it's a script or a bunch of open TOFU files
that have the same coupled infrastructure together. But since technology is not static, the needs of the team slightly change over time, and so whatever you built is now out
of date. It now is actually causing a cost and a burden to the teams to continue to use it because it's too opinionated, because it's a wrapper on top of what was actually available in the world before, and so actually platform teams in a one to three year timeframe, cost usually if not well maintained overall, actually hinder engineering teams for being able to deliver stuff effectively, and then until they realize, oh, we're actually getting in the way.
We need to build a real product to actually deliver effectively, and then turn that around and extend the platform and improve it over time until it reaches this more effective long term vision of being more agnostic and being able to just like raise up very small amount but consistently across for whatever their user group is seems reasonable.
Yeah, it seems like a good way to at least reason about it.
If we're going to try to set up mental models and not going to use the word I almost use the word frameworks, but trying to not anymore.
I'll take overloaded buzzwords for one thousand anlycs.
Yeah, so, I mean this actually came up in one of the communities I'm in. We're trying to define what the difference between code and configuration.
Is and it's a very blurry line now for sure.
Right, And so my point was exactly as yours was, Jillian, which is like, it's just a word that helps convey some sort of point that we're trying to make to help get us from where we are to somewhere else. We're just using the vocabulary we have to try to convey our thoughts and the most conducive way we know. And of course different people understand those things to be different and apparently also have very strong opinions on what
those things actually do mean. And if you try to challenge it, like you know, doesn't matter, they do sometimes.
Not too happy about challenge.
You know, they associate themselves with who they are based off of the tools they're using. And if you said, oh, you're just a configurator, you know, that may drive a stake in some people's you know, personal perspective of themselves.
Yeah, for sure, that's that could go very deeply into a whole other topic. But a lot of people in our industry, they do tie their professional identity to the tools, like I'm a terrorform person or or I'm an antiable person, or i use AWS and it's one of the things I try to share with people whenever I talk to them about their careers is don't tie yourself to that technology.
Tie yourself to the problems that you're solving, because in the course of your career, those technologies are going to change, but the problems you solve will be the same. And it's a very subtle but effective trick to making sure that you stay relevant for additional career options over the years.
That's why my title is not Biopearl software developer anymore. I don't know what it is right now, but it's something that it's not biopearl. It's not catalysts.
I don't know.
I don't even remember what any of the old biogragmatics frameworks now, but they're all pearl based.
Nice. How could you let that go?
I don't.
I don't know. I don't know.
Yeah, I would think most people will kind of. I guess if you're talking to like newer professionals, I could see how that could be a problem.
I don't know.
I don't think people in the field for a long time would associate themselves with the particular tool, but I could be wrong about that.
That Pearl is one of those, like the old school.
Pearl for like a long long time.
Oh yeah, and they'll still argue with you today about how great it is is and how humanity is doomed because we've moved on to something else.
That was the first programming class I actually ever took. I had no idea what they were talking about, and the professor just launched into this whole Pearl versus Python thing, and I was like, I just want to program some robots, Like I don't even know what you're talking about, but it stuck with me.
You know. Twenty something years later, here we are.
It's worring. You were going to chime in on Pearl.
Oh well, there is a joke there.
There's like some OCR that converts artwork into text, and it was like a terrible OCR program, like it did not do a very good job at all, but if you threw an image at it, some famous artwork, and it would generate just arbitrary tax which was like periods and semicolons and dashes and ad signs and every literally everything had generated was a valid Pearl program.
It's great gobbledygook, but Pearl will take it.
Pearl is the honey Badger of programming languages.
I still check in on like the Pearl six News sometimes and I'm like, ah, such hope, such hope, and such despair and like one small space on the internet, it's fascinating.
We'll get that when we get half left three. So let's talk about artificial intelligence and how that factored into the Dora metrics this year.
Because oh so excited for this part, I get to disagree with the whole thing, the whole one.
I read, well, never let that lack of knowledge be a reason that you can't argue a point right.
Lack of knowledge never form my opinions.
So in looking at it, this is a card graph to read, I found the graphs to be challenging.
So anyone who's actually looking at the report, what you have to remember is that at the bottom access it's often like zero point one point two point three, And what it really means is percentage of the total population or the actual percentage increase overall comparating the options. So really it's just often ten to twenty percent for the different areas.
There's the elites are you.
Know, somewhere around ten percent, and the lowest minority is also ten percent and is about twenty to thirty for the mid pieces. I don't know if that's the grapher looking at well, but I found personally the UX on the graphs not to be the best.
Yeah, that's that's very confusing. So I'm going to go with the text up above saying that eighty one percent reported they're shifting their priorities to increase incorporating AI into their applications. What's not clear is, well, I guess given the scope of the audience, we can assume that eighty one percent of DevOps professionals are incorporating AI into the develop services.
I think it's actually businesses.
Is the total population that they're talking about here, Okay, eighty one percent of businesses are trying to incorporate AI into the final product that they're utilizing. I think there's also a different metric for the amount that are trying to utilize it as a tool to do some sort of software development. But I'm not sure that's totally highlighted there. I don't know which one the eighty one percent was.
I'm so totally curious what the thing was that Jillian found that also Viamilly disagreed with I am just waiting in suspense here.
So well, so now.
I'm looking at the report and I can't find the sentence.
But I thought that it was something like, you know, AI adoption was kind of less than predicted, or AI significance was less than predicted, And I found that to be completely opposite, especially in drug discovery companies, where everybody was like, yeah, whatever, Like we've seen these before, right, Like Microsofts came out with their AI that was for healthcare a few years ago that was supposed to revolutionize everything, and it revolutionize like nothing pretty much, and so on
and so forth, and that has absolutely not been the case with these kind of with the newer models, with the chat ChiPT the cloud ones haven't gotten to try out the new reasoning ones yet. But in particular, the AI seems to have this kind of scary good knowledge
of how chemistry works. And it's great because my brain is like equally divided in half on this opinion where it's like, well, the AI sort of understands language and human language and that's what it's trained on, and our representation of chemistry is also humans creating language to represent something. So this also makes sense, but it shouldn't it shouldn't make sense the way that it does. And like the results that you get, everybody you know, just looks at
and says this should like this should not work. This is so you know, this is so scary good what the AI can do where you give it like a string representation of these compounds or these drugs or whatever, and you can take it and you can like start asking the AI to generate you Knew drugs, Like you can say like, hey, here's you know, I ran this experiment and here's all of these compounds that docked against
this other compound. Make me, you know, make me fifty new ones that are going to do like you know, that are gonna be comparative or something like that. And then you can also with these bigger context windows, you can throw like all kinds of public data sets at it. You can take open targets, you can take the therapeutic like there's just a crazy amount of data that you can incorporate into it. And it seems to be doing really, really well. And so that's my very narrow view of AI.
That is the only thing that I actually care about, which has nothing to do with this report. But I found the adoption to be much greater than was expected across the board, Like I had somebody say like, oh, you know, it was just this toy that I was playing with a couple of days ago, and now it's my whole job.
So I just find this very interesting.
I think that what is going to.
Be making our medications and I don't know how I feel about that it's coming.
I mean there is like I always had this perspective of humans make mistakes, and I think we forget that and we assume that if we design an AI process to handle something, that it has to be perfect and realistically it just has to be equal or better than what humans are doing, or even worse if there are
some other trade offs. And often when we evaluate a process, there's a very common aspect for engineers who say, oh, we're looking at the code or the output and it's not actually one hundred percent.
Correct, so we should reject it.
And I don't think that's necessarily the right mentality to have here. But what you're messaging you're talking about, Jillian, I totally agree with. I think outside of the software engineering space, the usage in built in product that are being built or the end product can be hugely valuable.
Almost everywhere.
I think, as far as the report is concerned, we should look at the impact of the product the company is creating versus the adoption by the software engineering teams, and so I think overall, what we are seeing and what the report says is that everyone's optimistic or most people are optimistic about the usage of AI for doing the development, doing software development, using it in the teams. But when it comes to the actual product delivery, success, performance, durability, reliability,
that actually decreases with the usage of AI. And I think this actually makes a lot of sense where we're employing very knowledgeable, inexperienced engineers, a lot of them to build us our products, and so of course we can.
Do it faster, but the outcome of what we get is going to be worse, and it's.
Going to be the case for I think a very long time unless we overcome the barrier of making truly more intelligent lms, and the nature of lms, I think is incredibly limited there in.
That regard, I think the lms a little bit they're going to have the same problem specifically for writing code that the package managers do, because when I've tried AI is out for writing code it like it mostly works, but it'll mix in different syntax versions of different of
like different programs. So like if you're using like PANDAS for example, that's kind of a common data science library, and they like to change their syntax quite a bit, so you'll get like different you know, different changes from different versions and things like that. So I think that'll be kind of the next I see that as sort of the next big change that will have to come along for the coding AIS to really take off.
Yeah, I mean what I've seen is the is sort of two shotting the output where you keep a collective dock of some of the learnings that you want to
have passed into every code generation you do. So, like, if you know there are differences in PANDACE, what you should do is have a doc that says and you can use your package manager as the source, like you know, only give me output that matches recommendation based off of the versions that I have or specifically say Panda's you know whatever, the version one point three five, and then it will help to make sure that it generates output
that matches this index from that version. And if you don't give that, then of course it's going to pull out the most relevant thing based off of the problem that you've asked, which may not be in the version that you're utilizing.
Well, I think that's going to be interesting with the newer models having such large context windows.
Is that I want for PI.
Charm to reach the point where I set up my interpreter and then the AI like it's just like this, this is the interpreter. These are all the versions and things that we should be using. Just stick with this the same way that if you have like the code Complete or anything like that set up, it takes from the It takes from the dock strings and stuff of the functions of the libraries that you're using. So I kind of want for it to do that.
I think that that's one of the hidden benefits of using AI for specifically for things like writing code, is you'll give your chatbot or whatever you're using a prompt and then it comes back with an am answer and you find scenarios like that and you're like, oh, okay, that's one hundred percent my fault because I didn't tell
you all of the constraints. And for me personally, I've found that it's helping me to become a better communicator because I would have the same problems interfacing with humans. I would say, hey, go do this task, and they come back and I'm like, ah, well, okay, I forgot to tell you it has to run on you know, Linux, or I need sixty four giga member. You know, all these little details that I should have communicated up front,
and so the same mistakes apply. Is just more rapid feedback for me to understand that I wasn't clear in my request.
Yeah, it's my fault.
I should have mentioned that we were using a sixty four bit operating system here instead of a thirty two bit. I mean that's a that's a new invention, I know, like just last year.
So I can.
I'll you know, got.
Confused here, but yeah, for sure that is exactly what's happening.
And that was some of the looking at the task reliance on AI here seventy four point nine percent of people responding or using it for code writing, sixty two for code explanation. I use it a lot for that.
What do you mean do you like highlight your code and then say write me a doc string or what do you do?
No, No, I'll copy some code into it and say what is this thing doing? And then it comes back and tells me, especially in like whenever you know it's a language I don't work with very frequently, you know, if I'm helping someone debug arrust application or whatever, you know, I'll drop that in there and say what does this thing do? And it breaks it down.
See if you if you get in that situation, I almost recommend taking the next two steps, which are first then ask it to generate unit tests for it if they don't exist. Where the unit tests are the documentation for the code, and then tell it to rewrite the code so that it's simpler and more understandable. And you have the unit tests as well, so that the next verst netlooks of the code knows.
That it what it does.
Oh wow.
And also you have the unit tests that were generated and validated regressions for the previous version, so you know the new version doesn't break either. And yeah, it's extra steps, but it reduces the loop there.
Damn level up, Warren, I go, I got.
Some good ones here.
So I think the doc where you actually have a bunch of recommendations in that you keep including its context
over and over again is one of them. Another one is that we're going to very quickly run into this place where LMS are trying to generate code in languages that is not optimized for like the code the software languages we've created, we're all created by humans to do human related understanding things, and over time we're going to if we lean more on LMS, we're going to get to the point where we actually want an LM focused language where it will be able to better understand how
to generate code that does the correct thing, understand the context of what code has been written and documentation around it in order to do the right thing, Which means we're going to end up creating software languages which are more less understandable for human and more understandable for machines, until we get to a point where what we're writing is almost completely not understandable by us at all.
But I think that's still a.
Little ways out, but we're going to get new software languages for sure that better incorporate this this concept.
Yeah, and I think that's the big like, that's the basis of every sci fi AI takes over the world story ever, right, is that AI starts improving things until it's beyond the point of us being able to understand what it's doing.
Yeah, but I mean look at like there's very few of us. I mean I did do this in my academic career. But you go look at the bytecode that was generated or the assembly by the assembler, and you don't like you are looking at that and be like, oh, yeah, I know. I think shift like three registers here, that's clearly a crypto you know, hashing algorithm obviously with a bunch of jumps, and this is making an HDP call. You see the register here and and the requests in the socket map.
Ya of course, like I know exactly what that's.
Doing, right, binary in your head.
Already looking at the minified javascriptsss for websites and you're like, oh, yeah, I know exactly which file this came from and what the stack trace was. When this is going to no, it's just like random letters that make no sense, you know, with parentheses and periods between them. So, I mean we're already at that stage. I think there's just another one above here, which is between using LMS as the input. They're still not the input to software development. It's gonna
but it is going to change. We're going to change the languages that we're utilizing so that they match better with our non organic coding partners. I remember, like years ago when the browser wars were like like at their peak. We were as a group of people and we were talking about, you know, different browsers and everybody was watching for their for a favorite, you know. And one of the guys in a group he says, a real man just uses curl and parses the HTML in his head.
But it's a fun one. But see that's why, like I'm a little bit surprised that I don't like the the lms don't do better with code because it is still humans like creating models out of language, right, I mean, like the code isn't completely that, but it can. You know, it's close enough. It's as close as chemistry is anyways.
So why do we get these really good results in some of these fields that have these representations of data that at least in part use something that kind of looks like a language, and not so much with the code.
So someone's going to call me out on this because I am for sure not the expert. So there's that caveat, and I'm.
Sure that's nobody is the expert that's the thing nobody knows.
Every meeting I'm in, we're all like, why.
Is it doing this?
Like that part doesn't get Yeah, I mean, if it gets cut out of the podcast, someone will definitely call me out. So handling human speech in written words has a very single directionality to it, forward directionality as far as linear goes in time, you sort of read the context, you get the result, whereas software development has a need for the output to be sort of understood from a huge context of a single class, an object oriented or multiple functions, et cetera. And so there's at least a
bi directionality things going forward and back. Like by the end of function or a file of which you have some source code, there's maybe some braces at the end, etc. Which aren't just based on the previous lines. They do impact what should have been written almost and same across files. Whereas you could have a many page book where each page in the book only follow from the previous one, but it's software development is more of like a choose your own adventure.
Some of those functions are used outside.
Of the context of that file, and so there's a lot that goes into it that isn't just the same as natural language, and so there's actually different models that are being used in order, like built up and design and how we're actually orchestrating the consumption of that data and the neural net that we're creating to process that data is fundamentally different depending on the output. It's much closer to say, creating an image than it is a human language, like a written text.
That's interesting, Yeah, you're right, all, but the probability space is like much greater with code than it is with human language, right, Like if you look at how the AI kind of takes thing, it's part of a very simplified explanation is that you know, it sort of creates like the probability that the next character in the sentence will be this thing, and it does that sometimes. And with natural language, if people speak king, I mean, I don't know, but I don't know that the severb and
documented anywhere. But I would guess that it's much much smaller for natural language than it is for code. You can investigate your own adventure.
For you can understand a lot more with mistakes i'll call them in human natural language, things that don't make sense, sentences that run on or curtail or miss the predicate, whereas with code it's going to get executed, and so there is some fundamental difference here. I don't know how it is in chemistry. I always can't speak to that. It's possible that the models are highly both fine tuned with that in regard, or maybe it is just the obvious result of a bunch of natural language as well.
It lends itself well to that. Whereas but I could imagine experiment diagrams or state diagram transfers I forgot they're called in chemistry, you know, it can convert from a couple of different molecules to different other kind of molecules like that. You know, of course, of say, a chemistry book could be challenging to get right, like it could definitely swap from talking about one kind of molecule to
a different one later. And where a natural language that may not totally matter within the context of running experiment, are actually designing a product that someone could use, Uh, would not make sense at all.
That is always a little bit freaky about the AI doing things, especially with like medicine and drug discovery, where you're like, listen, one compound is safe for you to take the mirror image of that compound is going to kill you.
Like, and these are you know, they're the same thing.
They're just they're just mirror images of each other. We've got to make sure that the AI, that the AI gets this. Sometimes. I know the ones that I've experimented with have just been the general AI models. I know, like new ones are coming out all the time, right, and I'm sure there are going to be ones that are more targeted, you know, specifically to certain fields like drug discovery or astronomy or you know, whatever the things are.
I have to know, now, does it often misdate getting the el chairo and ar chylal molecules mixed up, or is it like does it know that it should mostly be l chiral.
I don't know, because I haven't been running those types
of experiments. That was just kind of the easiest thing that I could pop up with in my head to say, these are the stakes for the AI getting things wrong, and they can be very simple switches, right, Like this has happened in human labs where there was I don't know, there was like a type of medication given to women during pregnancy for nausea that caused a bunch of birth defects and it was just a chiral compound, so your chances were, you know, like almost fifty to fifty if
you've taken it things like that.
So I don't know, Sorry, I lost my train of thought there. I don't know what I was going on about.
I mean, this was always so rivetting for me that the molecular compound equations could result in al chiral and ar chylal compounds. And in the human world at least like ninety nine percent of things that we encounter are all l chiral and they're all mostly harmless to us, Like the archylo versions of them, which almost don't exist in nature, are all like incredibly poisonous for us to consume in any regard, causing instant death.
And I don't know, this is just always so interesting to me.
Chemistry is pretty wild. Like chemistry too.
I'm trying to think of there's anything left in the in the report that we haven't talked about.
There's one thing I want to bring up because I found this to be really interesting. The drivers of adoption for AI number one, marketing number two, for GO bureaucracy number three. What if our competitor implements AI before us, and I find those interesting because I think those are
all the exact wrong reasons to implement AI. Like none of those say, oh, because it's going to help us to live a better product to our customers, which I think is should be like the primary goal, well to do to do that at a at a profitable margin, and that's just kind of the end result.
And people tend to, you know, drill down more and get more into the particulars of what they're doing on their day to day.
But I think that like if the.
But you can do all three of those things and not make a better product for your customer the way that they're stated here. Anyways, it could be.
It could just be screwing around with AI, and it couldn't you know, not mean anything.
I think there is right now. I think what's happening is it can be.
It's always a challenge for you when you're marketing your product to explain concisely what the value is that you're offering so that you capture your customers to come and utilize your things. And because it's such a challenge, I think a lot of people are jumping to just saying that AI is included in it because it is being used as a proxy for all the things in which
AI could hypothetically be doing to make it better. But then there is this sort of death spiral where that encourages the use of AI in the product to do those things, even though you don't fully understand the value it's offering. And then you have to say that it's using AI to do that thing, because people are concerned that if you're using AI, the output may not actually
be what it should be. It could be making more mistakes than you would expect if it was well coded and validated, and so you end up with this death cycle.
And I think that is certainly part of the problem for sure.
Yeah, I think especially on these experiments where they're large enough scale that you're not going to want to replicate them with people. So I see a lot of literature mining projects where it's like, we want to know all the literature that's not available on whatever this new cancer, okay cancer, And I mean, I don't I wonder how much it would cost if I was going to be like,
I'm going to download three thousand articles dust them. I know how much it costs for the AI to do it, But you know, just them with the AI, I'm going to ask it a bunch of questions and get some results. How much would the same experiment costs to be doing with people.
I don't know, but.
I'm going to guess, you know, I don't know how many how many papers can you really read before your brain turns to mush. I'm going to put it in about ten, so you know, you need a good number of people to do that.
And then I don't know of anybody who's doing that kind of experiment, so I don't know.
That's always like the interesting thing for me with like the marketing and the more like wiki style experiments is that I can't really can't really have a control group there, not one that I'm paying for anyways, not going to do that.
I mean, I think it would be and it would be interesting if you're talking about trying to come up with like real conclusions based off of having downloaded or
consumed all of those articles. Because if we look at what a human would do, like let's say intelligent software engineer one of our viewers, right probably write some sort of script to go and curl all of the documents, get them downloaded, and do some sort of rejex matching to find words in the document that you care about and then utilize that and investigate each one of those
more specifically. Right, you know, you start like a very high level approach to filter out the information you don't care about into only the parts you do.
And I think that can help a lot.
Now we're actually talking about consuming all of them and coming up with a collective, unified epiphany on the topic. I'm not sure that the AI would be able to do much better job than basically regurgitate some parts of the individual documents itself. I don't think it's going to be able to pull from a second corpus of like sets of epiphanies you can have and like do a match, right, you know, I read these documents and then like, you know, there's six possible conclusions you can come.
To, and you know, pick one of those.
I think is where humans are still generating new ideas, whereas we've limited the LMS on what they will actually generate. If they generate stuff from not the documents, we call it wrong and usually it is. And if it generates up just from the documents, then I think it's losing some sort of insight that we would get even if you took the human friend brain approach.
I think that's very true. The LLM is not going to innovate the way that humans are. Like presumably you could get some very smart human who looks at all those papers and says, this is what they all missed, and then we'd have, you know, some crazy new scientific thriller novel that the punch of nerds reading papers or something.
I don't know.
All right, it feels like time to do some picks. Okay, all right, Gillian, welcome back to the show. What's your first pick?
I don't know.
So the show got me thinking about neuroscience again.
So I'm going to pick the book The Man who Mistook His Wife for a Hat by Oliver Something. But that's the title, pretty ditinctive, so you can you know. It's on Amazon, I promise. But it's really interesting because a lot of the experiments in neuroscience that are done you can't you can't just run around doing.
Those two people.
Okay, we have like ethical committees for stuff like that. So instead what happens is that you know, people who have suffered certain brain injuries or had strokes or some you know, like something like that has happened. They are very extensively studied, and then we say, oh this, you know, these parts of this person brain got knocked out, and then this is this is what happened. So then you get very interesting cases like you know, somebody saying, you know, this is my.
Wife over here is actually a hat, and you.
Just you have all kinds of crazy stuff that goes on when you start messing with the human brain. So that's that, and then you can learn all about why you should not be experimenting on people.
Seems like a worthy lesson to learn.
It's important why mess with the human brain?
There may be some data points. We open up a couple of history books that provide additional context. There why that's a bad idea?
There are there definitely are really really maybe I don't know, just throwing stuff out here.
What about you, Warren? What do you got?
Yeah? So I didn't pick a book this week.
My pick is going to be QCon San Francisco, which should be happening this week. My my CEO of authors is actually giving the only security talk I believe at the whole conference, which is titled security or Convenience?
Why not both?
And so if you if you're there and you have the opportunity to go and see it, I highly recommend it. There's a lot of interesting insights where innovative companies are focusing some of their opportunities and what is almost now considered legacy.
Right on. Nice cool for me. I'm going to pick a book that we we were talking about before, just before we recorded this episode, and I think it's with the conversation we had about AI. I think it ties into that and we'll have some relevance there. It's a
book called Trust Me I'm Lying by Ryan Holiday. You may know him from The Daily Stoic, but prior to that venture, he was a marketing professional and did a bunch of different things in the marketing world that he's not proud of, and this book is his admission to doing those things. But it's really interesting to read to understand how corporate marketing works and the things that they're willing to do and will do to promote their brand. So yeah, trust Me on Lying by Ryan Holiday. Super
interesting read. It will forever change every commercial website, ad and blog posts that you ever read. And do you guys hear that clicking in the background? Is it just me? I do Okay, I think that's.
My computer, right, n yeah, that's good.
Yeah. Well sometimes it's just in my head and then I ask and everybody stares at me awkward, going uh no, dude, you just go over there in the corner and do your thing, and I'm like, okay, all right, Well with that, we have an episode. Thank you Warren, thank you Jillian, and to all the listeners out there, thank you for hanging out listening to our rants and ramblings and misdirection.
I hope you enjoyed the show. If you did or if you didn't, you know how to find us on various social media platforms, so give us a shout and let us know. We'll see in the next episode.