Chris Cooney on Exploring Data Evangelism

Hello, data aficionados and tech enthusiast. Welcome to the first episode of season 8 of the data driven podcast. I'm your host, Bailey, your delightful AI guide through the fascinating world of technology and data. Now, before we dive into the data laden depth of today's episode, we've got something rather special for you. Brace yourselves for our new theme song, hot off the silicon press, entirely AI generated. Yes, even our theme music has joined the generative AI revolution.

So plug in your headphones, turn up the volume and let's give it a listen. From trends to bikes, we light up your nights with inside it up loud. Let's get this party started. Well, what do you think? A symphony of zeros and ones, or should we stick to human composers? Feel free to drop your thoughts on our socials. Today's episode is a treat. We're joined by the brilliant Chris Cooney, a maestro in the realms of data, observability in production

systems, and developer advocacy. We'll be delving into the intricacies of keeping an eye on your systems, the art of data observability, and why developer advocacy is crucial in today's tech landscape. So grab your favorite cuppa, get comfortable, and let's get data driven. Hello, and welcome to Data Driven, the podcast where we explore the emergent fields of AI, data science, and machine learning, and, of course, data engineering, because without data engineers,

the world would stop revolving. And with me is Andy Leonard, my favoritest data engineer in the world. How is that for a new intro, Andy, for season 8? I like it, Frank, and welcome to season 8. Cool. Cool. Yeah. So I'm gonna tie in our guests, at least geography, with the theme of this season 8. And my promise to our listeners and viewers, is that we will not disappoint people like Game of Thrones season 8 did, and,

our guest is nodding. And, as folks know, a lot of, Game of Thrones was filmed in and around Northern Ireland, where oddly enough, as as the coincidence would have it, I have a a family history there going back, well, 2 generations from me. But, our guest today is, Chris Cooney, who is a software engineer, SRE principal engineer, and he is now the head of developer advocacy at Coralogix.

Hopefully I pronounced that right. And he's worked on everything from embedded systems, for controlling industrial battery units on the UK power grid, and to payment processing systems, that process, up to a 1000000000 a year, and just helping shape technology strategy, with 100 of millions worth of cloud and on premise infrastructure. Welcome to the show, Chris. Thank Thank you very much for having me. I'm super excited to be here.

Awesome. Awesome. We had a great chat in the virtual green room, but, so tell tell us a little bit about yourself. You're currently located in, Northern Ireland now, but you you your entire career, from what I can tell, spans kind of the UK. And you were at I see on your LinkedIn that you did one time work at Sainsbury's, which if memory serves, because I used to live in Germany and I would go to UK quite a bit, that's a grocery chain?

Yes. It's a it's a retailer. Yeah. It's the 2nd largest retailer in the UK. So my background, I've been engineering now for, oh, gosh, like, 11 years now, I think. I think I'm starting to get some gray hairs now. The, the the the gist of it is started out the application level Java, back then, then moved into React engineering because I realized I couldn't design a front end to save my life. And then, I

real after a while, I thought, well, how do I run this thing? So then I moved started moving into the DevOps and SRE side of things and started reading a lot about SRE developer experience, the what was emerging then, which was platform engineering, which was slowly slowly coming to the forefront, and then ended up replatforming, got really into the whole Kubernetes space, and then started thinking really heavily about, well, how do I keep track

of all this stuff? And after a few roles here and there, I went into engineering leadership. And, while I was in leadership for a few years as the principal engineer, I was responsible for there was, like, 22 different teams. They all had very different portfolios. And I was just trying to, understand what each of them were trying to achieve and then maximize that outcome. I realized very quickly that the thing we were lacking desperately was,

observability. And so when I started to look around, I spoke with Ariel Asarath, the CEO of CoreLogicix. And, if you have ever or will ever speak to him, you'll know he's a very compelling guy. And I pretty much signed up signed on the dot. I was like, let's do this for a 100%. I'm I'm ready to go. And then I've been there for 2 years now, started out as the

advocate. And we the past few years have really just been about understanding what the what the what advocacy looks like both in the observability space, but also at CoreLogic specifically. We have a pretty good picture of that in terms of content, tone, speaking, that kind of thing. We're hiring more, and we're growing that advocacy function to be much more proactive and outreaching and have a lot more

fun with it. And so, yeah. And and and, obviously, the reason for being so excited doing this podcast is the thing with observability. You know, most people will say, what's the problem with observability? Most often, the the thing that comes back is cost. People say, like, it's just so expensive. But, actually, like, if you deconstruct that slightly, really, the problem is data, volume, and how to manage it

because that's what's driving the cost. And so then you go back and you analyze the behaviors and understand what decisions we made as an industry that drove that. And all of that has got me really into the data space now to try and understand better what we can do both as engineers and consumers to make sure that we feel we're getting the best return on investment for every dollar we spend on our observability. So it's fundamentally a data problem, and that's why this podcast was so

exciting for me. Oh, awesome. Awesome. So, I'm gonna go was excited to talk to someone who's in developer advocacy. I worked in, event what we used to be called evangelism, for Microsoft for a number of years, but it's the same it's the same thing. Yes. What's fascinating is I think one of the key challenges of developer advocacy or evangelism was always the traceability. Now there's a number of problems. Right? But but one of them is the traceability of a a particular advocates activity

to actual revenue. I think that was always that was always kind of a something that I know Microsoft struggled with. Right? Yeah. And I remember when they when, you know, you know, this was well, 2011, I interviewed and they were like, do you have any questions for me? I was like, yeah. Well, how was an individual evangelist tracked? Right? Yeah. And the answer to that was a really good

stump speech by the hiring manager. His name is Dan. Shout out to Dan if you're listening. Was he goes, you know, you can't how do you track someone's individual activity? Right? At this time, Vista was still fresh in people's minds because, well, if you're a Windows evangelist, you can't blame you on Vista. Can't play Vista on you. Right? Okay. So how do you how do you kind of work

that? And how do you how do you message that? How do you how do you track that from, you know, inception to purchase? And it was a very interesting kind of thing. I never look at it the same way again because on the outside, it looks like the funniest job in the world, and it is a fun job, but but there is a business side of it too. And, Andy and I actually got involved in developer community, and that's actually how we met through the efforts of another, evangelist. His name is

Andrew. Okay. So, like, we got involved in developer community. At the time, the focus of Microsoft Evangelism was was building up a kind of a community of folks, user groups, and such. And it's fascinating to see how evangelism has evolved over the years. Yeah. Do you wanna talk a little bit about that? Because it's not. User groups are still important. They're still a thing, but it's actually, I think, a little different now. So you you mentioned writing and things like

that. What what what makes up a the the the offerings of evangelism these days? Sure. So I think the early days of evangelism were where this reputation for it being remarkably fun job came from. Because as is often the case, people are quite idealistic when something's new. So when evangelism came about, it was like, we need to be able to talk to engineers. What do engineers like to talk about tech? We're gonna hire people that talk about tech, and

that's their job. And that was that was the line of thinking, and then that spawned, you know, YouTube series and and and various different media that that that came from that. And then what happened over time is, like, as as it often does, economics came into play and was like, well, how what's the return on investment? Like, great. This guy just gave a talk on some very deep corner of Prometheus querying or something, or or, you know, Python. And what do we

get for that? And so now what's what's what is still actually happening, I think, is, organizations are transitioning away from a model where evangelists just talk about open source tech, and they just, quote, unquote, connect with engineers. And there's something of a pipeline in place there where they they they are if you like, the toppest of the top

funnel. They're right right at the top of the funnel. They are talking about open source technology or something that people will find interesting in general, And their goal is to talk to people often. Now when I speak to our advocates, the the the goal is to talk to people who might be interested in the product. They're not there to sell the product. They're not even there to market the product. They're just there to talk to those people and be like,

okay. You're all interested in the observability space. For example, for me, here's a talk that's useful. Regardless of whether you're gonna buy CoreLogix or not, here's what's useful for you from an engineering perspective. And, normally, what I'd like to do is 90% independently valuable information, 10%, sort of, the by the way, I work at CoreLogix. This

might solve some of the problems you've seen. So I give a talk, all over the place, which is kind of all about the summary of it is basically, like, how you can get the most value out of your observability spend. That's the kind of the the essence of it. I do not sell CoreLogic at any point throughout this talk. Yeah. But at the end of the talk, I say, by the way, if you're looking to cost optimize, CoreLogic has some great tools. It might solve

a problem for you. And so I'm I'm I'm the pre product marketing person if you like. And that's kind of of ad where where I see advocacy at the moment. Yeah. That's one of the parts of it. And then there's the community building and the content creation, that kind of thing. And in terms of content creation, you know, the word content really has only been a a descriptor for the kind of media that

we're creating, for a few years now. It's a relatively new word, And I think that the ability to create compelling content has is is becoming further and further to the foreground because the impact of one good video is, like, you know, it could be huge. It could be nothing. Could 2 views or you could get a million. You don't really know. And a lot of companies want to throw those dice. And I think that's where it's interesting. Well,

you're fascinated by your by your that's okay. I'm fascinated by your division of, topics. They're 90% just trying to help Yeah. And, you know, share with people from your experience, which is impressive. It's vast. And, you know, doing someone who's who's come through the field as the field has matured. And I I think that's very easy to overlook. Yeah. You know, how you have to as a developer,

especially, how you have to continually shift gears to grow with the technology as the new technology shows up and understand, okay. This is the problem it's trying to solve now. And that gives you a little bit of a, you know, a projection maybe or or a feel for maybe where it's going, a little bit of predictive

analytics type thought. Sure. And then knowing that the company that you're working with, that the the platforms y'all are building and and making available also address some of the current problems and also some of the predictive ones. I like that 9, 8, 10 mix. I I find that makes for a compelling talk. And when when I'm sitting in one of those talks, I don't feel like I'm being sold something. Yeah. And No. And I I understand that. Yeah. What's funny is that,

I've given the talks, and sometimes salespeople say that was very clever. And I said, what was very clever? And they say, you know, you you, I I you know, the sales pitch was there, but I didn't feel like I was being sold to. I was like, because you weren't being sold to. I wasn't I'm not trying to sell my job. I don't I'm not measured on sales or revenue. I don't get, commission. Nothing. Yeah. And so,

what I measured on is actually, I have a personal KPI, which I really like, which is if I give a talk at a conference, I measure the number of people that come to the the the booth that we have at the conference and just ask questions about the talk, whether they're interested buyers or not. In other words Nice. Like, people are interested and engaged enough that they wanna come and talk to us afterwards. Like, whether they're buying or not is not my problem.

But what is my problem is did I give a decent talk? And that that's usually a pretty good metric because the ones that I felt have gone really, really well, and then no one's come to the booth. And I've spoke you know, there's been drinks afterwards, whatever, and they were, oh, you were on stage. What were you talking about again? Like, no remembrance. And the ones that I felt were kind of

okay, 30, 40, 50 people come to the booth. I I I just I'm I'm struggling to field all the questions, and then I have the drinks afterwards. And everyone's, oh, I saw your talk. It was brilliant. There were you know? So my own sense of how good a talk was is pretty off, basically, last time in the audio. So so so that that that's a nice little KPI for me personally. Yeah. But, yeah, the com yeah. Sorry. Go ahead. That's okay. I was just saying that's a great KPI. And I just like

one quick follow-up, and I'll shut up and let Frank talk. Does anybody ever you mentioned the person that said, you know, that was clever. Does anybody in the audience ever provide feedback and say, oh, it was a big sales pitch. You ever get that? I so sometimes when the talks are very, very short, you know, I only have, like, sort of 10 minute lightning talks are different because I think it's a different head space. But when it's, like,

a 10 minute, 15 minute slot. In the past, not so much these days, I'm better at it now, but a mistake I've made in the past is allowing that ratio to slip from 90:10 to kind of 50:50. And I think that I've come off and they've gone, you know, it would have been nice if you focused on the open source stuff a bit more. And so but I think that 90:10 ratio, even the people that say at the end, they noticed that that's 10% as a sales pitch. They go, well, yeah, but 90%, no one was trying to sell me

anything. So whatever. That's fine. You know? Yeah. And I think as well, you just so much of it, like, engineers try 2 things that engineers are often, like, numb to now is 1, recruiters and 2, salespeople because they are just everywhere. You know? Right. It's very difficult. So to not if you if you seem like a recruiter or a salesperson, you just become part of the background noise of an engineer's life. And so so my goal is always to make it really clear. I actually when people talk to

me at the booth, I'm like, look, we could really solve your problems. And I can see them, like, go, oh god. Here we go. And I hold it quick. Just been saying, I don't work in sales. I'm not sales. Don't worry. This I have no best interest in you. Yeah. And so that's that I think from a from a from a sort of logical business perspective, that's a really powerful trust connection that you have with with engineers that,

that businesses can use to get their message out. That's the economic side of it. And from a, from a just sort of human perspective, that's the really fun part of the job. It's just like finding something you're passionate about, being in a room with people who are also passionate about it, and then having a giant conversation. You know, and if you're a bit extroverted like I am, then being the center of attention is is good as well. It's always nice. So, yeah, it's it's it's it's wonderful.

No. I think you said it best when you said it's the top of the funnel. Right? The tippy top of the funnel. Right? Because Yeah. One of the ways I've I I I've been I've had a number of roles that kind of dance between I've always been on the more evangelism side of sales when I've been in Sales Works. Right? Where I do create content. Because I gotta create content anywhere. You kinda just once you get the content creators bug, you have it. You know what I mean? Yes. And

and to your point, you know, I don't when I make a video, I don't know. Is it gonna get 5 views or is it gonna get 5,000? I don't know. I really don't know. I haven't been able to figure out that, you know, try as I might, try as I try try to figure out some kind of algorithm for it. I I haven't really cracked that, and it's it's also true for, for speeches too. Like, I totally get

it. But I think the best way that I've used to explain evangelism to non believers or advocacy to non believers is that, you know, salespeople go to the person in the corner office and ask for the deal. Mhmm. Evangelists warm up the crowd. Right? So they basically cubicles knock on the door of the corner office saying, hey. We need this. So that way when the salesperson does land, it's a warmer it's a

warmer call. And and that's something that, I think that when you're talking to salespeople, they do get that notion of, like, you're the warm up act. Right? Yeah. Certainly, all the salespeople in in CoreLogix were a bit confused by advocacy initially. And then when you know, one of the things you'll know you'll you'll all know this from working at trade shows and and conferences, there's the you have people at the booths, and they're trying to snag people and

be like, hey. Yeah. What are you interested in? And then suddenly that changes to 30 or 40 people just turning up to the booth to ask questions. It's like it's so much easier for them. And then they kinda they okay. Right. No. I I get why this is good for me. Like, I the leads come to me. I don't have to, you know, I don't have to go out and find them. So that so so that that yeah. I think the salespeople are certainly the ones

that I've worked with now get the value. But I I I still think we're only doing sort of 30, 40% of of advocacy and evangelism and let us hire more people, and hopefully, we can do the other side of community building and all that sort of stuff, which is all gonna be new for the company as well. Yeah. And, you know, it's it's very avant garde that your company is even doing it. Right? Like, you know, it's it's it's it's something that even large companies don't

really do. Obviously, FANG, to a certain extent, does. And I posit that because of Guy Kawasaki's work in the eighties. Okay. Right? He was the he was the first person that I'm aware of that had the title of evangelist. Right. Apple I think a big part of why the Macintosh has this cult following, and and I know all the Mac lovers go, don't call us a cult. Right? I know. I have a I have a MacBook. Right? But, it's because of his work, like, you know, 30, 40 years ago. Right? I I

don't think that's a coincidence. And I think that, when Microsoft wanted to have the uptake of dot net, they reinvested heavily in in evangelism. And I

think we saw the the fruits of that. So you no. I mean, it's it's one of those things where I've noticed that it kinda ebbs and flows. Right? It's like, you know, there's a the tide is up, everybody's all into it, and then the tide kinda goes away and people kinda Yeah. Down on it and they reorganize and they get rid of it. But I think that it is it's a field that I think has still maturing, I think, to your point.

Right? Like, you know, because how do you define it? So there was something going around on LinkedIn where a lot of folks who were advocates tagged in an avocado emoji on this. Yes. Yeah. Yeah. Could you explain that? Because I only know part of the story. Do you It's an Amazon thing. Oh, okay. Because Advocate and Avocado kind of sound similar. As far as I'm aware, that's the that's the depth of it. I mean, I think it was changed with Amazon, and then, people just started to do

it. I did it for a while. I think I don't know if I still got it now, actually. But, but, yeah, it was just a nice thing to signal you as an advocate. And the advocacy community, I will say, is is singularly wonderful. They are such lovely people and, you know, all of them, I've been with competitor companies. You know, sat on on the front row ready ready to go up. And if someone's nervous, like, because no one's we're not we're not competing

with each other. We're just giving talks. Everyone's super friendly, relaxed, talk blah blah blah. You know, everyone's like, the and the the way the community is, that that is one of you know, and you make friends pretty much instantly, with with with all these different people at the booth. So you suddenly you go to one conference. You just make 5 new friends. And, you know, 2 of them are have been doing this for 20 years, so they're just fountains of absolute

wisdom. You know? And so it's the community actually makes a massive difference for it. And that avocado thing I might it's just a sing a signal, like, you know, we're we're in we're in the this weird little club with you. And and, you know, what is ostensibly a very strange niche, a soft niche of of software engineering. My boss, when he, when Ariel interviewed me, he said, congratulations, Chris. You're an extroverted software

engineer. You're one of the 3. And I was like, yeah. Because it's it's, you know, it's it's a it's a weird space. And not that you have to be extroverted to be fair, but it it does kind of help, you know, going out and meeting people and shaking 200 hands and and answering all the questions and being on stage, it does help if you get energy from that, within which I definitely definitely do, thankfully. Well, let's talk about the data part because I think that's the part that,

so and I think it I think it also dovetails too. Right? Because there's certain you mentioned k p a KPI that you have. Right? And there there's clearly data that you have to collect as part of just being an evangelist to show your your and I'm sorry. I keep using the word evangelist. That's just the old habit. It's fine. Fire department. But, there's clearly data you track. But what's the core core core what's the

core of CoreLogix? That sounds really weird. But, like, what's the core of the business? You mentioned observability. Right? And observability implies data. So talk to me about this. You mentioned Prometheus. So explain a little bit. I'm I'm giving you, like, a ton of questions together. Sorry. No. It's okay. Yeah. Yeah. No. I get it. So You're drinking Diet Coke. I have the monster energy drinks. I see. Okay. Right. Yeah. Okay.

Something of a higher caffeine content, I imagine. So Right. The the the the gist of CoreLogix is this, full stack observability, processing logs, metrics, and traces, and we have a security offering. The the the essence of the platform can be distilled into, taking decent data science principles around how to manage your data and then baking them into observability. One of the problems that we so we actually did some investigation a few years ago,

about how people are actually using their observability data. And we found some really, really surprising statistics. Like, for example, 99% of index data in in a in a in an elastic search, or an open search cluster, for example, is never queried. So 99% of it is is ingested into the cluster and then just never touched. And and it's one of those things where it's like, wait a minute. Like, why? And then we realized, well, it's not queried, but they're maybe

visualizing in dashboards. So I was, okay. So that's interesting. Then another one was, a large volume of data. I can't remember the exact number, but a large volume of data is only interesting for a single number in the log. This is primarily focused on logs. It's just one log has a latency field, and that's what people really care about. The rest of it is just noise. For example, the the average historical query length is a week, so people

tend to query back a week and not much further. The retention time is between 2 4 weeks, on average. So at the very least, we retain in high performance storage for twice as long as we need to, on average in 4 weeks, the upper end of that. When I do this at conferences, I say, hands up if you retain for a week, 2 weeks, 3 weeks, 4 weeks. And inevitably, there's always some poor guy with his hand up who's like, you know, we're on 3 months at this point, and I'm like, can you just tell me how long

you retained? Was that a year? And I'm like, okay. We would've been here for a while. So so so what we found in the industry was that there was this perception of we need to send less data. We need to, we need to we need to do more with less, I suppose. And what we found was, no, the problem isn't data volume as such. We have this data because it has a purpose. It has a function. It has a reason for being. We don't just generate

the data for the fun of it. We data we generate data because our infrastructure's becoming more complex. Our, the the solutions that we have to come up with micros microservices. You know, at some point, somebody had, you know, decided that they have a 200 user a month CMS, and they were like, we need 15 microservices to run this thing. I don't know when that, like, gripped the the popular consciousness, but it has become a big thing now. So all these practices drive up the

data. We need the data. So instead of deciding what data we're

gonna chop and change and and get rid of and that kind of thing, Let's say, how would we go about retaining everything? How would we go about keeping all of our data? And then the question was, how do you manage the cost? The answer is to be use case driven with how the data is stored and how the data is accessed. So as I mentioned, some data is only useful for a single number converted into a metric. Metrics are a fraction of the the cost to retain. So there you go.

You just you just shaved off the vast majority of that document's overhead, convert it into a Prometheus metric or a Victoria metrics or whatever works. But the thing is, we don't wanna we wanna retain data for a really long time because archiving and rehydrating is both expensive and painful. So instead of that, let's break the rehydration paradigm and directly query the data in your archive with no dependency on indexing or reindexing.

Some of our data is queried. Some of our data is never used. Some of our data we ingest just because we might need it in the future. Okay. So make different levels of pricing based on the use case for that data. And so that was the that was the, the pinnacle of the sort of the the the what what was distilled down into what became the CoreLogic stream architecture. So I find myself talking about that the most. There's a tonne of, you know, tracing and, database APM and serverless APM, and

q Kubernetes and so on in the platform. But, ostensibly, at its

very, very core, it's just some smart data science and data engineering principles, including the fact that we built everything on a streaming based architecture. So, rather than doing everything in, like, a batch mode or triggering everything from a database, we use Kafka and Kafka streams to process the data and make decisions in flight rather than sort of various intermediary storages that have IO bottlenecks and all the rest of it. Makes it very scalable and also very efficient to run.

It's less which means less expensive for us to run the platform, less expensive for the customer. So we drive enormous cost savings as well based on that. So all those things, there's a lot of information there, but all those things kind of, like, come together to form this platform that gives you all the traditional observability things that you want and some really, really

advanced stuff. And it's also possible to query your archive, which is actually an s three bucket or cloud storage in your own account, directly from things like you can query your metrics directly from your archive. You can query your logs directly from your archive, your traces, but, also, you can visualize stuff in dashboards alongside index stuff from your archive. So it's it's all about and the the difference is you wait maybe 2 seconds instead of a sub second rendering time. You know?

It's not much of a cost, but a massive, massive saving opportunity. And that's that's kind of where we've gone. Now we're just building this wonderful platform that's really, really fully featured and really mature. So interesting. Yeah. Absolutely. When it terms the, like, the the the record keeping of logs in terms of history, is there a difference between different industries? Like, there's certain regulatory things. Is it come up in digital forensics, I would assume?

Yes. So we have some really fun use cases. We have some companies that are using us. So as I said, the core of CoreLogic is just great data engineering principles. That's what our architecture is all about. And, that's illustrated best by the fact that we've got some companies that are sending, you know, maybe 40 or 50 terabytes of data through us a day. And they're using us purely as a transformation and analytics engine. They don't index any of

their data. It actually goes back to the s three bucket. In this case, an s three bucket in their account. And we transform process and analyze that data. This particular company is in the financial industry. So heavy, heavy regulation. Everything has to be retained. Everything has to be accessible. They get regular audits where they have to demonstrate. Like, someone will come in and be like, tell me what happened with this user on the 1st June

2022, you know, to have to have the data that far back. If they wanted to do that with any other platform, things would get very pricey very quickly, and we offer that archive query at no additional cost. It doesn't cost anything per query. It's purely based on gigabytes ingested. So they get basically enormous retention, enormous

scalability without needing to pay the cost. And that that digital forensics thing is a really, really interesting part because people think of observability as a purely DevOps or resiliency kind of discipline. Yeah. It's not. It it's it's all about understanding what you've got, data observability, measuring the freshness, distribution volume, and so on of your data, is within that observability realm. And it's possible to do it in the same platform provided that platform has been built with

those decent data engineering principles in mind. And so, yes, the data forensics piece, analytics sort of, you know, aggregations across 1,000,000,000 and 1,000,000,000 of documents to get some really some insights that would otherwise be hidden away. Yeah. Those are all things that we run into regularly. I I think it's really interesting that you took the approach of streaming first.

And I could see several advantages to doing that. Often, people architect data engineering, even near real time analytics, collection to be more focused on historical and being able to move the window around. And then they have an option that they kinda bolt on, And, you know, they're just trying to pound on the server Yep. Pull the server, throw it into a loop where every second is grabbing the most current data. And if you do it

that way, I mean, you're you all are nodding your heads. You you get it. It's you've gotta architect from the ground up, I mean, you know, to have that perform at all. But if you take that approach of reading real time data, to start with, that's your focus. It's very easy, I would think, or easier to expand the window and say, no. Let's look at the last hour, the last day. Yeah. And and more more

than that as well. The it's not just so so the streaming architecture, basically, as far as, as far as I see it, then I I I this may be more of an opinion than an engineering position as such, but the streaming gives you the ability to make decisions in flight, and it makes it really easy to perform transformations. And, as long as you're disciplined about minimizing side effects, it also unlocks, the

scalability. But it's the things that sit around the stream, which is so we have this concept of, source, stream, and sync. And, Yoni Farrin, the CTO of the company, and I believe his team kind of came up with this. But a source is like, you know, a a a the data source, the actual database, that kind of thing. The stream is obviously the Kafka stream that's processing it, and the sync is where the data ends up eventually. It's a pretty normal thing in data science

anyway. But the the key thing here is that none of our, sort of in stream transformations are reading from external sources. All the data is loaded first, pushed into the stream, and then written. And what that means is that the individual streaming processes can be horizontally scaled very, very easily. We can so what it means is that we could scale up and scale down

very, very effectively and very efficiently. That's part of that cost optimization on our side, but it means that that's a thing that, I think is really important to highlight because people often I worked in Sainsbury's for a long time and I saw many Kafka projects appear and fail. And it was because while they built the Kafka someone built some Kafka solution, They constrained it left, right, and center with IO and database sort of operations. And so when it came to

scaling it, it was like, well, this isn't the bottleneck isn't this. The bottleneck is the 30 or 40 database and Redis clusters that are sitting around this thing. So it's not just the foresight to build a streaming architecture. It's a foresight to minimize side effects in your architecture, and that makes it much more scalable and, ironically, much more observable as well because the the everything's just an in out transformation. Makes it a lot easier to monitor and maintain.

Cool. That's a lot that's really clever, actually. Like, the more you the more you explain it, I'm like, that's brilliant. Like, what? Why didn't I think of that? Just like I tell you, like, these the part of the reason I joined this company was because I had a call with I had a call with Yoni, the CTO. I had called Ariel, the CEO, and, I left both of those calls feeling like I was just like a chimp banging 2 rocks together. And I was like, right. Okay. This is where I need to be. I need to

this is this is how I level up now. So and it's I've learned so much since joining, not just as a I love the architecture. Sorry. I didn't mean to cut you off. I I love the architecture, and I love the patterns in both of those kind of appeals. I mean, but, you know, I I would imagine that the IO on the data store is critical. That's gotta be mission critical at that point because you you wanna catch all the data as fast as possible.

And at the same time, you wanna be able to serve that data as fast as possible. It's, you know, that would be the bottleneck, I would I would imagine, and I'm sure y'all have solved that at the, you know, tuning the hardware up. You know? Yes. Precisely. Yeah. It's it's it's it's it's a comp it's essentially, partially an architectural decision and partially an engine partially an engineering decision. So the architectural decision is, for example, because of the

way Kafka works, we can parallelize a great deal of the processing. So to give you an idea, a typical observability platform, one of the large vendors now, you can expect an alarm to fire. Let's say a metric crosses a threshold. It's anywhere from 2 to 4 minutes on the on the upper end before the alarm actually fires. Wow. That's a big delay. That's huge. It's enormous. You know? And if if you imagine you're a financial trading company, like, 4

minutes is, like, is is huge. You know? If you're if it's a security alarm, that's massive. And that's because what they do is they ingest the data, store it, normalize it, and index it. And then they trigger a series of processes. And this normalizing and indexing process only gets worse with time. That's an infinitely scaling dataset. What we do is instead is we do all the analytics upfront, and then if you want, we store it

in high performance index storage at the end. Otherwise, it just goes straight to the s three bucket archive or to the, to the cloud storage in your account. Okay. So I love the flexibility of that. I I do. I could see how that serves the architecture because, you know, often, 4 minutes being notified 4 minutes after, a hack starts, it's over. It's useless. It's some so we have this type of alarm called an

immediately type alarm, and I run some use cases with it where, a an IP address appears in the logs from AWS, web application firewall. We have an ability to enrich data as it comes in, so you can we actually look at the top 15 threat databases when an IP address comes in and we say, oh, is this malicious? And then the alarm was triggering every time there was a malicious IP, and the immediately type alarm was triggering it under a second. So it was,

like, firing pretty much instantly. And in within 18 seconds, it had invoked a Lambda function, which, updated the IP set in the in the WAF instance. And so it's like sub WAF itself does that in in over a minute. So the data was leaving AWS. Oh, that CoreLogic is in Amazon, actually. So so but but it was it was going to CoreLogic's being processed along this extremely complex

pipeline. The alarm was firing, and the Lambda function was being invoked in a 5th of the time it was taking for WAF to even measure the data in the first place. That's huge. Yeah. It's enormous. And

that's that's like that it to me, that's the best demonstration of our architecture. That's like Yeah. That that that's one of the reasons why it's just so remarkable. And and it's also it's a it's like I say, it's a lesson in data engineering. And when I when I go to conferences and talk about cost optimization, I find myself talking more and more and more about just good practice of managing your data, as opposed to

any secret observability magic source. It's always the way, isn't it? Like, a relatively new industry has a lot to learn from the adjacent industries and almost never does. It takes a lot of time to that passion. So yeah. No. That's very true. Like, there's a lot of good lessons to learn from, like you said, adjacent industries because we a lot a lot of the problems we're facing are not necessarily new. The context is definitely new, but the Yeah. Laws of physics are persistently

stubborn. Yes. Precisely. Precisely. Interesting. Well, are we at that point, Frank, where we're ready to pull up the questions? Okay. Alright. So we'll ask the pre canned questions. They're none of them are real brain teasers. Right? They're we're not we're not trying to be Mike Wallace, and I don't even know if anyone get that reference these days. Okay. But you can always ask chat gbd who Mike Wallace was. I will.

But in order to save the environment from hitting those GPUs, all these people hitting the GPUs, Mike Wallace was a journalist for TV show called 60 Minutes who was notorious for, you know, if there was a corrupt executive or politician, he was notorious for, like, sneaking up on him and asking them, like, while they're, like, getting groceries or whatever. Maybe even at Sainsbury's. Who knows? And saying, like, you know, hey. You know, why did you embezzle $5,000,000? Like, what's going

on? It's a bad day if you walked into your office and Mike Wallace was waiting. Just sat there. We could do with some of that now. I think we should resurrect that practice in the UK. We need more of my gualas', I think. Yeah. I think you're right about that. Yeah. So the first question is, how did you find your way into data? Did data find you, or did you find data? Data very much

found me. I as I said before, like, I was primarily interested in SRE, and then I realized, like, then observability became a real passion of mine. And then I realized, well, what is the biggest problem? Oh, god. It's data. Okay. This is okay. So that's that's how I ended up here. So what would you say is the favorite part of your current job?

Every time I finish a talk and someone comes to the booth and says, you know, I've been really struggling that for a long time, and that thing you said there has just given me a really great idea. And I that is, like, amazing. That's just just, like, direct dopamine to me because it's, like, engineer to engineer having a conversation and just solving problems is, yeah, brilliant. So, yeah, right now, that's that's the the best. Nice. We have 3 complete sentences.

When I'm not working, I enjoy blank. Oh, god. Is this just like what I do in my in my normal Yeah. In your spare time. Yeah. Yeah. Yeah. Spare time. So I Yeah. Well, whatever that whatever spare time. Yeah. Yeah. I I try and, I try and spend as much time with my daughter as possible. She's 10 months old and she's, just crawling and everything else. So

just absolutely fantastic. And, yeah, it's a level of, like, it's I'm tired of doing way wrong, and I'm a bit stressed, but, it's like the the the peaks of joy are just, like, unbelievably, like, incomparable. As one, I like philosophy, and I like, I play some guitar as well. So I've been kind of getting into that more as well. Cool. Excellent. Yeah. Dad, Frank, and I are both dads. We set up. And I'll tell you as a dad of, daughters, I have 3 daughters and 2

sons. And I was I'm the oldest I'm I'm the oldest of, like, 5 boys, so I had no clue about either daughters, sisters, anything like that. But, yeah, it was it's an it's an amazing experience. Yeah. And watching them grow up and my baby girl is in college right now at Virginia Tech. And so you I saw that look, and it's like, yeah. That's gonna feel like about a month from now when you look back when she gets there. Yeah. For sure. The days are long, but the years are short. That's the Yeah.

Yeah. Yeah. I have 3 boys. 14 is the oldest, 9 18 months. So yeah. It's pretty chaotic. That's great to hear that. Our our second complete sentence, I think the coolest thing in technology today is fun. I hate giving this

answer because I feel like it's everyone's gonna say the same thing, but it's it's it's gotta be Gen AI. Right? Like, right now, it's it's it's just like some of the thing. I so I, I'm half, Arab, and I've been learning Arabic for the past few years. And the other day, one one of the problems with Arabic is that, online, you basically find all the lessons in classical Arabic, but you'll never find lessons in what's called the dialect.

So, like, I'm Jordanian. So the Jordanian dialect is is is it's not completely different. It's very similar, but it's there's lots of details that are different. And I went on to the new chat gpt model the other day, you know, 4 o, and I, opened it up and said, I hit the the headphone thing to have a conversation. And I said in Arabic. I said, I want you to speak in Jordanian dialect to me, and I don't want you to speak in in in classical Arabic. And it responded

perfectly. And I thought Wow. Like Wow. I don't I don't even know what resource you went to to get this information. But, like, it that you know? And it was it was the the accent, like, wasn't right. Obviously, the the release in the new voice model, I imagine that might change things a bit. But just the grammar, the inflection, the phrasing, the slang was was all that. And I thought this is

like, I can't find this information. I've tried I've tried for for a long time to find this information in written form or in just a one place where I can go to get a, like, a full sense of Jordanian dialect, and it just doesn't exist. So so I just thought, wow. That's that's pretty pretty crazy. And that's yeah. So that's probably the coolest thing I've seen in a while technology

wise. I'm from the villages, so I should say that. Yeah. But Well yeah. But no. But I mean, language learning is one of those things where I I did take a couple of courses on Arabic, and the teacher was from Syria. Yeah. But everybody in the class were as was this was Jersey. Yeah. The state, not the island. And most of the most of the other, participants were from Egypt, and they would always argue over how to say things. Yeah. And as a non native speaker, I'm kinda like

completely lost. Oh, mate. You know? And they you even like, one of the things that's we we say Arabic in English, and we what we're essentially describing is a whole family of languages. Right. There are lots of Egyptians who if you ask them, do you speak Arabic? They say no. I speak Egyptian. And if are you an Arab? No. I'm Egyptian. And they because there's a 40% of the Egyptian language is is influenced by Coptic language.

So and and then if you go to, like, Morocco and you ask them, do you speak Arabic? They say, no. I speak a Tarija. Like, the I speak a and and and it's like everything, you know, even even, like, Lebanese, they say the Canaanites rather than rather than Arabs. Now there's obviously politics and things mixed up in that, but it's also embedded in the language as well. The Lebanese will use a lot of French, for example, because of the history there, but also because of the

it differentiates them slightly from from their neighbors. You know, in Jordan, they use a lot more classical Arabic in how they speak, but also use a lot of English in how they speak as well. Again, because of the history, but also, again, to differentiate themselves. They say different work, different letters differently, and so

on. So so when we say Arabic, you know, the the general perception is that it's it's a one monolithic language, and, actually, it's a very, very wide, the glossier of languages. And, yeah, I've I've very rarely seen resources acknowledge that. I've never seen a resource ever automatically talk back to me in Jordanian Arabic. I've never ever seen that ever. So,

yeah, remarkable. Mostly remarkable. That is cool. Yeah. So the last complete sentence, I look forward to the day when I can use technology to link. I am really looking forward to where the VR headsets get to. I worked in VR for, like, a week, and it was the Quest 2, so there was no HD pass through. But it was still pretty great, and I I I had to stop every few hours because I was getting headaches and

stuff, and I had to kind of, like, take it easy. And we I really just feel like we've got the tracking sorted, we've got the processing sorted. Almost everything's there. It's just the comfort factor that we have to work on. That's just gonna come with lighter and lighter components until it yeah. It was like the visor being made by a company called the MERST right now, and they haven't done a demo of it

yet. I'm very excited for that demo, but, I I'm I'm really looking forward to the day when I can just have an empty desk and and just put the visor on and I've got all my screens and everything that I that that's that's really what I'm excited about. Yeah. Absolutely. Like, the the sweat that you build up around the is is is is really a limiting factor. That Yeah. That a little bit of motion sickness, but it depends on the game you play, I found. Yeah. Yeah. For sure. So

share something different about yourself. But do you remember it is a family podcast? Something different about myself. I I boxed. Oh, really? Yeah. Not professionally or anything. Just, I go into it for the fitness to start with and then just got more and more obsessed with it. Greatly enjoyed it, and, and I only stopped recently because I moved house, so there's no gym near me. But I absolutely loved it, met some really wonderful people, and learned a great deal about myself.

That's a quote fight club. You don't really know yourself until you get punched in the face. And and you really do find out a lot about yourself. So yeah. Excellent. So Audible is a sponsor of Data Driven. Do you do audiobooks? And if so, can you recommend a good one? I do audiobooks

all the time. Nice. Whenever I run, whenever I walk the dog, it's either a podcast or an audiobook almost guaranteed. Recommend a good book. I'll do 2, one tech and one non tech just because life's too short for only tech books. For tech, I will say Team Topologies, I think is one of the books that impacted me the most when I read it and made me think about how people collaborate with one another. That or Team of Teams by, Stanley McChrystal is just, oh, amazing as well.

Both of those are kind of in the tech realm. Non tech, I would say The Master and, Margarita by Mikhail Bulgakov was, it's just I read it and I wanted to I read it. I've read about 5 times and I just loved it every single time. I think it's just wonderful. And, also kind of morbid book Crime and Punishment by Dostoevsky, I just think is, again, I read it and I was greatly impacted by it whenever I read it. I thought it was wonderful. So, yeah, kind of too dusty Russian writers there. But,

yeah, those are the, those are the recommendations. Well, that's the beauty of the time we live in. Like, you could get an audio book on just about anything. Anything. And listen to it just about anywhere. Yeah. That's the thing. That's the thing. You know, when time is limited, I can take the dog out for a walk, audiobook, and I'm I get to hear about the, you know, some crazy you know, some detail of the Napoleonic war while my dog's chasing a stick on the

beach. You know, it's it's like that level of convenience. You can't beat it. You really can't beat it. And on the as well. Like, no. It's wonderful. For sure. If you go to the data driven book, which I think Andy is testing right now, see if it's a DNS that works. It's working. You know, for 2 tech guys, we we have a lot of infrastructure challenges. We definitely need some SRE. What do they say about Shoemaker's children? Where can people find more about you and CoreLogix?

So the easy one for CoreLogix, corelogix.com. And there's, there's a whole host of different ways you can learn about CoreLogix, video courses, Mostly me, so I apologize if you're sick of the sound of my voice because it's gonna be a lot more of it, I'm afraid. And then, we're on YouTube. We have a blog and all sorts. That's CoreLogic. For me, personally, I'm mostly on LinkedIn these days. I have other social media, but I don't really use it. So, if

you just search LinkedIn for Chris Cooney, I think I'm the top one. I think that's my accolade now. But if not, if I'm not coming up, Chris Cooney, CoreLogic is guy. So that For me, you were the 3rd. For you, me, you were the 3rd one. Yeah. But, I mean, you're first in my mind and my heart, of course. But, I when I saw it when I saw the picture of the guy that first come up, I'm like, doesn't look like you. And she, I don't think you live in Miami. So so I wish. It would be nice, but I'll yeah.

Miami is awesome in the winter. In the summer, it requires a certain type of person. That's all I'll say. K. I I I struggle in heat above 25 degrees Celsius, so I imagine Yeah. Miami is probably not for you. No. I agree. Not for you in the summer. Yeah. But I love Miami. Big shout out to, Miami. I know a lot of folks who live there and love it. Noel and Bill are the first two, 305

people that comes to mind. Of course, Pitbull, the the musician, but that's a that's a more of an insight, I think, to my musical taste that people want. And with that, will it barely finish the show? And just like

that, we've reached the end of the first episode of season 8 of the data driven podcast. We've traversed the fascinating terrain of data, observability in production systems, and developer advocacy, all thanks to the insightful Chris Cooney. A big thank you to Chris for sharing his expertise and making the complex sound oh so simple. Now, a quick note on our new theme song. We know it's a bit lengthy, but fear

not, we'll be trimming it down for future episodes. Your listening experience is our top priority after all. As always, we'd love to hear your thoughts on today's episode. Feel free to reach out on our social media channels or leave a comment on our website. Don't forget to subscribe, rate, and review us on your favourite podcast platform. Until next time, stay curious, stay data driven and remember the future is data shaped. Cheers.