Mental will be is a silent pandemic. According to the who depression and anxiety caused the global economy over one-trillion u.s. dollar every year, it's time to make a difference. Learn how to enhance your life through a master class from Founders well-being. And my good friend, Sunday prowl on mental Wellness, visit Founders well-being.com / Master Class to enroll and enter tlj 24. A 20% discount be a better version of yourself and make an impact.
The ISA QB software architecture Gathering is the International Conference highlight for all those working on solution structures in IT, projects primarily software Architects developers and Professionals in quality assurance but also system analysts who want to communicate better with their developers. A selection of well-known International speakers will share their practical knowledge on the most important topics in the state of the art software
architecture. The conference takes place online, from November, 14 to 17. And we have a 15% discount code for you. Enter tlj underscore MP underscore 15 for 15 percent discount. If they want to get value from their data, generally data-driven value at the one to do that by applying analytics and AI in almost every aspect of their business. They need to utilize data for both aspects, and all touch points and all applications inside the company and outside.
If they have such a mission and they want to do that reliably resilient lie and do that at scale fast, then they've got to consider this dish. It's all about really are locking the value of the data. Hey everyone. My name is Henry Surya with Robin.
And you're listening to the technology, you know, podcast the show where I'll be bringing you the greatest technical leaders practitioners and thought leaders in the industry to discuss about their Journey ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our Journal. Hello again, my friends at my listeners, welcome to the technology.
Now podcast the show where you can learn about technical leadership and Excellence from my conversations with great thought, leaders in the tech industry and you're listening to the episode number 107. If this is your first time listening to technology, you know, subscribe and follow the show on your podcast app and on LinkedIn, Twitter and Instagram. And if you'd like to support my journey, creating this podcast subscriber, A patron at Tech. Did you know dot f / Patron?
My guest for today's episode is Jean meghani. Jama is the founder of data mesh Concept in 2018 and since then has been evangelizing it to the wider industry, including writing, her latest book, titled data mesh in this episode. We discussed in depth about the data mesh concept which is starting to become an industry. Trend nowadays, we started our conversation by discussing the current challenges working with
data such as the outdated data. Approach and why the current data tools are still inadequate, jean-marc then describe data mesh and why organizations should adopt it to generate data driven values at skill jean-marc, then explain the four core principles of data mesh, which include domain ownership data, is a product, the self-serve data platform, and the Federated computational governance.
I really enjoyed my conversation with jayamma learning the data mesh Concept in depth, which has been something, I would love to learn more More about and this episode taught me a lot about it. If you also enjoy listening to this episode, will you help share it with your friends and colleagues who can also benefit from listening to this episode? My ultimate mission is to spread this podcast to more listeners and I really appreciate your support in any way towards
fulfilling my mission. Before we continue to the conversations, let's hear some words from our sponsors definitey is the top International software development conference with an emphasis on coding architecture. And Leadership skills. The lineup for this year is truly stellar and features many Legends in software development
names. Such as Robert Uncle Bob, Martin can back Scott Hanselman, Franca subramanyam Carolyn honey, Alan Halep, Mary, poppendieck, and many other prominent names, including some of those who have also appeared in this podcast before the conference takes place online. So you can enjoy it from the comfort of your couch. We spoke to the definitey organizers and I'm happy to share that technology. You know, has got the 10% discount code for you. Enter the promo code, awsm
underscore tlj. When you purchase the ticket on definite e.com, here's the promo code. One more time awsm underscore, tlj. Depending on the time when you purchase a ticket early price is still available. See you there. Today's episode is proudly sponsored by skills matter. The global community and events. Form with more than 100,000 software professionals here members, can organize their learning experiences around the technology topics.
They care about most you get on-demand access to their latest content thought, leadership insights, as well as the exciting schedule of tech events running across all time zones. So whether devops our data science is your bus or you are fan of functional programming or all things Cloud, you can make real connections with people who share your Our interests head on over to skills method of calm to become part of the tech community that matters most to you.
It's free to join and you will find it easy to keep up with the latest tech Trends. Hello everyone. Welcome to another new episode of the package you know podcast today I'm so excited to meet yarmulke ghani. She was last a director of emerging Technologies in thoughtworks. She was there probably around 11 years, my nose. I'm not following her work when I was working in thought. It's as well.
So she was part of the technology radar comedy and always come up with all these emerging Technologies and recently, in the last few years, jean-marc came out with this concept called Data mesh. I think it was around in 2018,
if I'm not wrong. Since then, the data mesh concept has taken a surprise by many people and many people Rave about it. So today we'll be talking a lot about data mesh and I'm really looking forward to have this conversation with you Jama. It's a pleasure to be here and really thank you for that perfect pronunciation of my name. Mmmmm. Oh, surprise that. Okay, yeah. I would love to probably know more about your career.
So I always stop to ask my guests to share their career Journey or any turning points or highlights in their career maybe if you can share a little bit about yourself. Sure. Laughter I guess my journey is filled with detours and going to new places led by curiosity. So I started as a software engineer for the first 14 years of my career. I worked in Take RND companies, where they were building a technology product side. Dates distributed systems before cloud and large-scaled.
It's really systems. Was thing. Build monitoring and observability from scratch building steering systems, building databases that basically gets signals from critical infrastructure. Analyze those signals turning them into reports and so on, the full stack of what real world, this was so could look like And I did that on multiple operating systems, various laser is a feelings and HP Tandem and windows and so on, that really gave me a good Insight, bottom up to the technologies.
That I realize that a lot of Technologies today. Maybe start with web development or app development, and they don't get the opportunity to really like look inside the kernel. And how to do system programming, how to build product calls. I did all of that, which was awesome. And then, I took a bit of a detour. It went to a hardware for a little bit. I work for a company. It was building various Hardware
from scratch. We were building the firmware on digital pen systems and that was interesting as well. Again, a lot of great learnings, how to build embedded firmware and then, another detour I came to Consulting without works. That's we worked at the worst. We work on have largest skill. Exact you ssion. So that led to microservices and
again building large scale. I guess this resolutions with micro services and that ecosystem like did quite a bit of work in that space for quite a while and I was Excited about service mesh and kubernetes and all of the Technologies. That's really made it possible. And as you said about four years ago, I kind of started putting my nose into the data space to my surprise, the world of data was far from the agility and nimbleness and totally and distribution decentralization,
that was seen active world. It's slow. Moving is based on paradigms around centralization of data and middleman moving data around and up. It was very sad. Observation, to be honest. That I thought, okay, I could be the little kid who points the bit naked Emperor. And I don't like doing that even if I get attacked. So I started like talking about capping would shift the parents on. What are the pain points? Like wake up people? I am. Yeah. And they suggest really came as if I caused.
This is a question with some answers, some basic answers since then I've been kind of building it, evangelizing it. And as of two weeks ago, I decided to start A company to build a product, a technology deep take, kind of Dev product, developer facing products that it's going to make it. So easy to kind of work with data under the mesh principles to really show the world a different way of doing things and enable developers. Most importantly, while sounds
real, exciting. Yeah. I saw your post about quitting Pollock's job. It was surprising to me but I think looking at the opportunity of investing more effort in building the tools about data. I think that will be definitely crucial and will help a lot of people. But before we go into this exciting journey, I think. First of all, the topic of this conversation is about data mesh. And you wrote a book with the same title data mesh with the subtitle of delivering data
driven value at scale. So maybe if you can share a little bit, what did you actually see the problems when you were working with the data? You mention about some old ways Paradigm centralization and things like that. But what kind of challenges and problems when you work? With the data problem during that time, and maybe if you can give an overview, how data has always been approached in the delivery or in the day-to-day development. Yeah, that's so good. Way of positioning data mesh.
The problem that I saw was that we had an assumption that to get value from data and value being through Pi business intelligence through reporting, through training machine, learning models, or all sorts of analysis of data data needs to come on. It's modeled or raw and skit. Centralized that was a response to a siloing of data, in applications databases.
That really didn't allow cross-cutting analysis of data across town occasions, but that centralization that idea of centralization of had led to a very fragile, very slow-moving, architecture and bottlenecks for really getting value from data. So what are the points of fragility? That causes so much waste? Those are data Pipelines. So to centralized data from applications, we have this concept of ETL elt, you know, extraction of data for
application databases. I think it's curved the biggest clients and we can do it. Architecture is so intrusive, right? Because there's no contract. There is no abstraction. So you have constantly breaking very task-oriented job. Oriented a labyrinth pipelines are complex fireflies, moving stuff around and converting them putting the from one thing to another thing. And that is a very fragile and Causes a lot of waste.
By the time the data has popped out of the other end of the pipeline, The Source has moved on and you got problems to solve and then on the other hand you have this big bottleneck. So the assumption that there is a data team responsible for the data from everywhere, and they put it in a warehouse of Lake. It is a flawed assumption, in an organization that needs to move fast, leads to share data. More peer-to-peer, they become a bottleneck, the team of the architecture itself, become a
bottleneck. So you have frustrated users As data users, they can't find the data they need. They can't access it. They don't trust it by the time they to God made available to them. The source has moved on data people in the middle that honestly I have all the oven empathy for them in the world that they have been. Given this impossible task of getting data from people that have no intention of sharing
data. And giving it to people that they have no idea how they're going to use it and it's stuck in the middle like waiting. They do watching and troubling data with no ready purpose to be honest. They are under a The pressure and the don't perform and then the applications really don't have much of a visibility or even opportunity to use that data. They just put data how to put that in the databases and use it for their operational needs and they never in the conversation
in analytics. So they're isolated from real application of possibility of the mail embedded into their applications, but becoming analytical data users because if they've been got, just put a satellite from this data world. Yeah. So fragile Leti. The only time from day Attitude value, bottom legs, out of that are major problems that you see with the past part. And I think in my career so far, I've experienced things like for example, big giant database where you have when the database
like Oracle SQL Server, right? Then we move into the paradigms of data warehouse concept, where you have another set of tools. So you pump the data, from the oil, TP database into this,
analytical data base. And then in the last, maybe 10 years we moved to another concept called Data Lake where you probably Release raw data, put it there and from there you move into different data Maps, may be of different small data warehouse and now we move into this Cloud Model where we also see Cloud Technologies. Helping I'm familiar much more with like bigquery and Amazon, maybe you have red shift.
So if you see all these historical, the unique thing about it, like you mentioned the centralization, why people actually move more towards centralization rather than the decentralization part? Yes, easy to get started with right when your problem space is new. New or solution space is new and you require specialization. You've got a new set of tools and new ways of working with data. You can leverage your majority of organization can really push the responsibility into every
team. You have to centralize a specialized people and also it's easier to control and easier to get started with like we know right? Any startup start building an application or solution. We say like start with the money list, find your Market. It and then break it down because decentralized distributed systems are inherently complex.
So there has to be a pivotal point that the complexity of your environment increases to the point that centralized Simple Solution does not work anymore and I think the data warehouses or lakes or lake houses as an architectural Paradigm, not so much of an underlying technology. They've been suitable for this world. That the data wasn't ubiquitous perhaps we were capturing. And it'll from every touch point.
The data wasn't being used in every single application domain and having a centralized team and decentralized Technology stack to do with it was acceptable. But we don't no longer needing that world. We've passed that pivotal point of complexity. So you mentioned about this point about operational forces, analogy goal, divide where the application team probably just
dump data in the database. And maybe there's another set of data analyst data engineer, trying to get the data from those databases and put it in a central place and getting in. Sites. There are a lot of complexities definitely but I think in the last few years we also see a lot of advancement in data Technologies. So I think I saw in your book and presentation.
You have this one slide we're probably there are so many different Technologies. The logos are too small, why those tools still couldn't solve this kind of problem because it seems like there are so many advancement. Yeah, that's a very good question. I mean, the tools are solving these problems. Oh, actually, collide with an example. The nevertheless Yes, they are created for an operating model that at the end of the day is pipeline data transform.
Put into storage layer with metadata, or spring, 40ml on top and voila. You get value on the other end. So they have been organized around this very centralized pipeline model at a very macro level, right? If he's ooh Weld and if you start sprinkling, these tools to an overall kind of big picture meta. Picture, that's what they're organized to do with it. Organized to solve ingestion problems, that were organized to self big hairy part, line problems their flow.
Like those sort of Technology. They designed to solve the big data storage or parallel processing problem. So unless you change the operating model and that meta architecture, no matter how locally you optimize a particular solution, we're going to optimize and solve the really hairy centralized data pipelines.
You still going to have a Harry satellite data pipeline, you get a little bit better and maybe detecting the errors of connecting to more resources, But ultimately you're still stuck in the past Paradigm and those fundamental assumptions that need to be invalidated and need to change remain. In fact, people who are deeply in the data space and try to use these tools, they say they struggle and they suffer.
They've got this massive landscape of fragmented technologies that frankly work, really, with difficulty with each other. The work with them have to integrate themselves. That costs of integration looks of this technology to a meaningful. Scalable solution is very high. I mean, if you look at every vendor on that diagram, and if you go to their connector Pages, this business is built around, just custom proprietary
connectors to yet. Another day's associate allowed, a distinct lack of standardization is just mind-boggling in this space. So it's like the Tower of Babel, or like it's just falling apart in my mind, it just feels like nobody speaks the same language. And to a large degree tools are built to solve a very custom solution, and the foundation of this house is about to fall out of. So, you still have a rocket Foundation. I know, I can be very polarising as I describe this because, I'm
very passionate about. Let's fix the foundation. Let's rethink. Our operating model. Totally makes sense. Because he had the way you mentioned about cost of integration. I can imagine every data products that I see, you will see all these Integrations, then the more the better after you do it actually the in Essman the kind of like lockdown and you put so much effort and maybe money to actually move your
data. But eventually if you need to switch also the costs for you to move the data out to the different Technologies. I think that's so painful. So I've been into some of these kind of projects. I think I agree with you about that problem. So you invented this data mesh, right? So if I may describe data mention from your book, you mentioned data, mess is a decentralized social technical approach to share access and manage analytical data in complex. And large-scale environments
within or across organizations. So they have so many interesting topics but the first that I picked is actually you mentioned it as a decentralized socio-technical approach. So tell us more about this sure it actually started as an architecture because I am a technologist so I kind of apply the lens of technology to solve problems. So I saw this as really an architecture to organize how we decouple and how we break down these big problem of how I get
value from data. But very quickly, I realized as we know, Conway's law and just real life experience, technology and architecture mirrors and get influenced by the way, we organize our organizations and teams. So, very quickly, I had to like, self-correct. No, this is not just a technical solution. No. This is not just an object, or we've got to rethink the organization of teams. The modes of communications, the contract for data sharing between the teams and the She is like data product.
Owner was a new role that we introduced. So parents, it became a substitute technical. As in we try to find excellence in our Solutions involving the interaction of people and teams and the technology, some people say, always a techno social or is it Associated that I don't really care which one comes first, as long as they're both involved, hence the word. Thanks for sharing that. It seems like the Conway's losses. Really like true principle in many software.
Architecture, right? So I think database is probably one of it and you mentioned that it is an approach to solve complex and large scale data problems. So does it mean that not everyone will need to go to data
mesh since the beginning? Yeah I think at this point in time I mean I answered usually this question by saying well at this point in time, if you don't have the organizational complexity, if you're late for laid out so where does model is not a bottleneck for you, is the centralized data thing is doing a great job and Everyone's happy. Well, why introduce a concept? That's rather complex. It creates kind of system complexity.
So yes, the short answer is. Yes it's not for everyone at this point to tell her I mean future technology advances are sinking or approaches are process advances in a way that bootstrapping with data mesh is as easy as bootstrapping or even easier than its centralized. So then at that point you said well, there's missions for everyone, because the maturity of support of the environment has reached level of Shorty.
So you have shared all these problems challenges that you saw before, and you came up with this concept. But for those people who are already in this state of complexity of dealing with their data, either, the data architecture pipelines, and things like that on organization, that is very large scale, maybe Global where they have all this data challenges. If you tell them, what are some of the reasons why they should consider moving to data mesh?
So maybe in business value or maybe in some kind of more impact, value-driven, kind of a benefits. Yeah, just simply if they wanted subtitle of my book, if they want to get value from their data, generate data driven value and they want to do that while I am applying analytics and AI in almost every aspect of their business.
And they want to you to do that. They need to utilize data for all aspects, and all touch points and all applications inside the company and outside, if they have such a mission and they want to do that, Reliably resilient lie and do that at scale fast. Then they've got to consider this fish, it's all about really are locking the value of the data. So, let's give a real world
example. If you are in a particular part of the business, let's say I use this example in my book of this, but if I like company recorded, a think, it's a Digital
streaming company. And if you have a team, whose job is really to create immersive musical experiences personalized for every moments of every person in the world depending of what they do. That team constantly comes up with new hypothesis and how to use data about music and artists and listeners and their behavior to create a more immersive experience. More personalized that moments in life, every one of those hypotheses they require discovery of the data and access to the data.
So are they going to be more successful to be able to discover and get access to the data and even ask people to provide the data to data is not there. If they were working in a peer-to-peer fashion, or are they going to be more successful? If it was a centralized team in between all parts of the
business? So as an example, if the playlist team that generous Emerson playlist, want to create targeted music for people, if there are doing cycling or running, are they going to be more successful to go and talk to teams that are taking care of partnership with cycling platforms directly and say, look, we need to see what people are responding to it when they're on that pallet on this. I suppose as an example, Oil or are they successful?
If they say to a middleman and data broker team and say look I have this hypothesis. So as all of these other teams that you need to Now put on your centralized backlog and plan somewhere and get me to the data that I need, that doesn't scale. So imagine your organization emerging the missions and the values that can be enabled through the data and see if you have bottlenecks that need to be addressed.
Then if you do then think about dating message though, You describe this use case is very interesting because yeah, maybe not all organizations are in this state where you have data and you do Discovery and maybe shape the next set of data that the application produced, again, do hypothesis and maybe analyze. And then again, reshape the data over and over iteratively. And then maybe one day you will come up with a new insights and maybe new business lines as well.
Because the data has transformed so much with the scale of the discovery, and also the scale of the hypothesis, that the The team does. So I think that's really a very interesting concept. I haven't experienced it myself because I haven't work in this kind of organization, but thanks for sharing that. So let's move on to the principles. I think, when I read all these data mesh concept that you share, you always come up with this four principles of data mesh.
And when I read all of them, I find it interesting because you kind of like use other kind of framework or maybe approach from application development and mix it into the approach, dealing with data. So maybe if we can Go briefly one by one, right? The first one is you call it. The principle of two main ownership and this is something like applying domain-driven design to data. Maybe if you can tell us more about this principle. Sure you arrive. Your observation is very
correct. That I wasn't as clever as creative. I basically contextualize the things that I had seen working in 24 years of my career in complex environments in operational systems and say, let's contextualize and let's apply them to the world of dating. As we've seen these principles work before, why shouldn't they work with this bottleneck? If sole purpose? Not only? So, the wind ownership is about, it's the same as what domain
driven design. Meant at the Strategic design level applications, which is you have this smaller business domain oriented teams and groups of people that are collaboratively across, functionally working to solve business problems with technology. So there Ultimately responsible to develop applications and software to enable that business outcome, but they're also responsible for using and sharing data out for analytical
purposes. For application of machine, learning model, again, back to the Show streaming, if you have a team that is working on your player application and their job is to give the best digital experience to a user. That's playing music or playing and liking it recording whatever the interactions are. You are also responsible
Responsible as a teen. Well, augmented for house, but new roles and team members for sharing that data in a way that data can be used to directly by some sort of an analytical work load, and that analytical work of might be a machine learning model, that is being trained by your data and some other data from other domains or it could be the reports that we are producing in terms of errors and anomalies of this application.
So we can improve it over time. That's the core of it is that break down the responsibility of data, sharing around the seams of organization. So you have this infinitely, scalable model. As you introduce new domains, you introduce new data sets that there was no mates can use and share and give the responsibility of this charity
to people for analytics. Guessing it for this, kind of cross-cutting use cases to people that are capable to be responsible for it because they're so close to it. They understand, they know what they say about. Don't give that responsibility to someone. One Downstream that actually doesn't know the domain and spirit hard for them to keep cognitive load of knowing all the domains is their team's heads. So that's the first principle. The way you describe it?
It sounds intuitive, right? Yeah.
Why not? But we came from the traditional approach where we have centralized team, they have to understand all the domain within the company organization, understand the data model, the evolution of all the data, and try to put them in one central place, but I think your approach here they are principal putting in To a domain ownership means that the domain team themselves is responsible for, not just the operational data, but also the analogy call data part where
probably they will transform the operational data and responsible for sharing them for analytical purposes. Well, and since they are the experts of the domain, they are probably the best person to come up with that kind of a data model. So I think it's really intuitive after you explain it. There's also one concept that is on this principle, right? Where you mentioned.
That the data pipeline, Is now is not a responsibility of a central theme, but now it becomes an internal implementation of that domain team itself. So maybe if you can describe why this is so important. Sure, well, if there's a fish is really successful. The way I had imagined that they're no longer be any data pipelines. I think if you go to the macro level, macro view of your architecture, you really shouldn't see data pipelines anymore. Between these domain between the data products.
Just concept that we can introduce it, the next principle. So if you think about pie, that is what's the purpose of them pipelines or job oriented task-oriented, computations that happen on some input from the data and transforms and put it on some output sink and you repeat these task oriented process until the leader gets transferred. In a mode that somebody can use it. Usually there is structure around looking. We need to extract information. We need to cleanse them with a to model.
And the usually done in between the scenes of the data outside of the source outside of the destination somewhere in between. So, Taylor is completely challenges that concept, because we're no longer working this task oriented kind of environment. We working on this value oriented outcome oriented environment data product re into the environment. So then the job of the transformation is really an
implementation details. One of these data products in one domain, it's not something that happens in between. And it's really falls into the similar principle of micro services that we had in microservices world. This Enterprise service was an anti-pattern, right? Because we want to localize logic and computation and complexity inside and boundary of a contract abstracted within a service, the Enterprise service bus wasn't doing that.
So, we came up with this idea of smart, endpoints and dumb pipes. So your pipes are super down there, just transform data and your Our inputs are smart because they implement the logic behind the apis of that. Led to kind of more API thinking world. It's the same process. So these pipelines are like the Enterprise service bus analogy.
If I choose and they have the same challenges, so we've got to break them up. Followed the pieces that are relevant, should feel where they should be known, they can be implemented as Parkland if you want to within the implementation or other ways. Again, very intuitive. If you again compared with the application development, right?
So yes, be has long. Been an anti-pattern So the same concept in data, if you want to connect to different data products, you should not create like a complex data pipelines. It should be just maybe transferring the data when transferring the data. This is also another concept under this principle that you mentioned, there is now probably not a kind of like notion of one source of truth anymore about the data. You will have multi maybe shape
of the data multiple copies. Maybe it is in the domain team themselves, or maybe it's already copied to the other consumer of the data where they will use it for their the use case. So why is this the case? Why normal single source of truth? I thought, like in the data world, people love to have a rest this single source of Truth. Where is the data that I can trust because single source of truth? It's not the real thing, it just doesn't exist in real world. Okay?
So let's unpack that. I don't intend to claim that we are becoming irresponsible about data and you will have contradictory copies of the data lying around and have Kind of concerned those that's not the aim. The aim is we still want to be able to get a consistent View and understanding of the data, but we want to do that in a way that it doesn't slow movement. It doesn't snow value generation. It doesn't become a sterile
source of Truth, very quickly. We want to do that in a way that we support the chaos of reality of organizations, their different teams will be different spaced, they generate different bits, And pieces of properties of the same entity, but those properties come from different sources with different Cadence. So we want to embrace that almost complexity chaos, but yet create a system that gives the same outcome that the single source of Truth wants to give,
which is if I search for information about customer even though that information comes from different places, I can understand a particular snapshot of the customer at a point in time. Has a set of consistent values.
That's why I challenge this notion of single source of Truth. So they admit when people don't actually read or understand, it has a lot of constraints and disciplines built into it. For example, the data that data nodes provide or read-only, they never change their temporal, they have two x terms by temporal.
So at any point in time, there are streamed across different nodes in a way that is some data arrives, new data arrives Upstream, the downstream notes that are For transforming that were copying that and transforming it into a new shape of the data they get notified. They have a responsibility to either react on it and generate a new slice of the database
pointy toilet or not. And then also this image provides a means of stitching is probably same Concepts like a customer that comes from the call center versus customer. That comes from the cover set, for the punishing concept of the customer, can be stitched together by the consumer because there are links that are It's between those systems. So between those little product. So there are set of constraints.
There are set of operational disciplines like this policy, meaning cage, and by temporality and immutable data, but still results in the saying, outcome of a single source of truth. But it's designed for inherently complex, business model, and operating model, if that makes sense. And you mentioned it as a most relevant copyright. So maybe you don't get the latest up-to-date. So it's like the concept of this asynchronous or maybe eventual
consistency. Maybe you just need a snapshot of a data at certain point in time. And yet the consumer will decide, okay? I just need this kind of data instead of always getting the latest. Let's move on to the next principle which is a principle of data as a product. And I see that you are applying product thinking to this principle. So tell us more why. It is important to treat data is a product.
Yeah, I think of Mind shift that needs to happen when we put data sharing and serving it, Lighting the experience of people using data as a first-class concern, is also an antidote to the first verse of all the problems are the first principle, so domain oriented data ownership. One can imagine can lead to Beta siloing on the player domain, but the data that I need to improve my application. So why should I care about sharing that and be responsible for consumers, which adds a ton
of work. So this has a product try to incentivize people to share that data. Be part of an ecosystem that is generating value through the exchange and through data sharing. And again, put some discipline and constraints in place for that to be done effectively. So defines such roles of people that are responsible, for that define success, metrics for data
as a product. Define, an architectural consent, if we define usability characteristics around discoverability address, but it's like all the things that make the experience of the data user, really easy, really delightful. Those need to be translated into structural Architectural Components that are built into this data as a product. So then you have kind of technology that needs to shift and change.
So, yeah. So in short is an antidote to the problems that are always from the first principle and also really focus on again getting value from data who gets value from data that users do. So, let's put the first. I like the way you explain, why the concept of a product is up here because you mentioned for a successful Products, you need these three attributes which is visibility valuable and usable and I think traditionally again, we just read data.
Okay. This is just a data you go and figure it out so sometimes it's not usable. So sometimes maybe the query language is different. The database Technologies different because we all have this polyglot database Technologies or maybe it's so ancient right Legacy Technologies. We don't know how to deal with it. So I think if treating it as a product we also need to think
about the usability aspect. So I think that is definitely The key when I read about this principle, there are also a new role that is being created because of this concept of data is a product. You mentioned about data product owner, and data product developer. So not all teams have these roles yet. Tell us more about the importance of these two roles.
Yeah, so if they support it becomes a thing that we are creating maintaining operating evolving retiring with it's useless and nobody use it, then they have to be people and roll. Rose to take that responsibility on and it's very unfair. I think it's impossible to say to app developers who's working consumer or very different personas, like they're serving the end-user to say, oh now from here, all of you also have this
other responsibilities. So not only your serving these end users that are interacting with your player application pressing buttons and make sure they have a responsive app and all of those great things but also half of your day you have to face around. And face this data analyst and designs in different domains who want to use the analytical data that you're generating, you've got to serve them to it's
impossible. You turned up two bosses like because service to purposes at the same time so if there are super humans that can do both jobs. So be it. That's fine. Maybe it is possible to kind of share your time and split your time that way. But nevertheless you need to have a very explicit responsibility and accountability for that. Part of the job.
If there are people that are appearing, like the data product, it will offer and adaptable, preparing and of collaborating closely for the two different people playing these roles and that's fine too. Yeah, so unless these kind of idea of the data Pride doesn't happen out of a good intention, we have to allocate space, we have to empower people have to keep people accountable as follows for hence, the roads. Yeah. And also not to mention the
skill set. They are totally different Technologies, different paradigms. I think it's very difficult to find people who can Master both application development and also data engineering. I think for That to be also usable as a product that you mention about all other usability attributes, like discoverability, address ability. It should also be understandable and it's trustworthy as well,
and it can be interoperable. So imagine if you have multiple consumers, then they want to access your data, but they do have set of constraints on how they would integrate with your data. I think interoperability is also a good concern that the data domain owner should think about. And that's why I think having a data product owner that maybe can Define the set of Of requirements and maybe it's kind of all usability concerns. I think that is key why these
rules exist. So let's move on maybe to the next principle which is about principle of the self-serve data platform. I think this is also interesting because you kind of like apply platform thinking to data mesh, why we should have a self-serve data platform. Yeah, I think it's an obvious one, but let's go deeper into it and say, maybe answer it. What is this little platform? So, when we think about It's the roles of Platforms in general
platforms. Are often shirt kind of infrastructure on top of which you build domain specific solution. So they often like the Mets diagnostic infrastructure that Empower other teams to build Solutions on top. So the valley horizontal not really bear to go so much. So in an organization, that's implementing data mesh. There you have this domain teams, they have active faults in them. You have data product, folks in them and they're doing their
daily job there. Daily job should be focused on delivering value based on the outcomes of that domain. Their daily job should not be focused on metal work, creating kind of the way diagnostic pieces of technology that they need. So, I think platforms as mean to empower autonomous teams to lower, their cognitive load to do what they need to do. More easily. They're wonderful. They're necessary. In terms of data mesh, the data platforms, many of them exist or
lots of Technology out there. They're built To give basic tools like you want story short? I will give you a storage. I can provision that bright. You want workflow processing. That's why like I'll give you that but the to eat slow devil for a developing the product. So we need a new layer of the platform that really takes data product away at least a to mesh that I defined envisions treated as a first-class concern and
hides away. Those details of, oh, I need a storage partnered with a, see that and it gives His life to a completely new concept, this new concept of this Quantum as it did product. So the job of that platform, the reason I put it in was making it feasible for independent, domain teams to do data work. And some of the attributes that you mention of this data platform is that it should be autonomous interoperable and domain Gnostic.
So I think one of the challenges when I was working with data related stuff, also is that? Yeah you have all these tools like you mentioned, but bootstrapping all these tools takes a long time. You have the setup clusters, you have to maybe install things dependencies and on top of that, then you have to write code, you have to maybe understand the source, the sink and things like that. And then you have to write the pipeline's itself and then
deploy it, and things like that. Yeah, it takes a lot of effort just to come up with a very simple Pipeline. And I could imagine having this kind of self-serve data platform, maybe something like the UI console, where you can just go login. And, you know, click, I want this data from this source and move it to my sink and then it just creates everything for you. I think that will be a perfect scenario where maybe some of the tools, not yet, catching up.
But yeah, hopefully, one day we will reach the experience for the data engineer or maybe for business people, they don't even need to care about dealing with data Engineers themselves. So, you mentioned about this concept of self-serve data platform because one of the challenges of building this platform is that you will need to make it agnostic. So, I think this is probably one of the challenges because we have so many different data Technologies.
So how Should we think about building this platform? So that it becomes agnostic because we have so many Technologies, right? We have so many different shape of database Technologies. Yeah, I mean it depends the agnostics city, I guess of the platform and its independence from underlying kind of Technologies the level of it
depends on the appetites. It's an organization that how independent they want to be. So I think what we need is interoperability between the different technology, so if it's still the solution on top, And the solution requires data from across two different platforms. There is a level of interoperability that I can access data across two different clouds or interest to different technology Stacks.
So at minimum we need and the year that creates that interoperability even if the vendors themselves are not incentivized to do that. Right. Now when it comes to underlying infrastructure agnostic again, I don't think it's meaningful in all organizations because you end up with a like most common denominator of the Features that are available on all the platforms. And that's not ideal. That's a lot of work and very little result so don't know if it needs to be completely
agnostic. But we have to have the pieces of it, that enables interoperability and movement moving from one to another and remove locking as much as possible. And those pieces are usually around cross-cutting concerns, right? How do I manage security? How do I be able to automation? So that if tomorrow I want to move to a different Set of infrastructure, my processes are automated and not hand-cranked.
I can kind of through automation facilitate the movement, much faster, that's the way I think about this being Tech agnostic. As opposed to a nice layer on top, I don't think that's really realistic speaking about cross-cutting concerns. So this also touching on the next principle. The last principle which is principle of Federated
computational governance. It's quite a mouthful to mention that but it's taking care of all these cross-cutting concerns are Mention things like for example, security policies and things like that. You are kind of like applying systems thinking to this principle so that we can govern the data better. So it share more about this principle because to some people this might be hard to kind of like understand. Yeah.
It's a mouthful already. And if I could sneak in another word, I probably have called this principle of embittered Federated competition covering. But I think Marti Fowler would have not Not posted my article, right, did that? Yeah, so I think that the concept is really is again, I don't to do two problems that arise from the previous principles, which is we need interoperability.
We have now this despair it sets of data products domain oriented, their own team, their own Cadence their life cycle. How can we apply set of concerns that need to be standardized across all of them? And what's the best way to go about defining them? What's the best way to go about implementing them? Observing and enforcing them that leads to this principle. So as an example, if you know to have secure data or you need to have high quality data, let's go
with high quality. So server quality data definition of quality and then enforcing quality and Sherry quality data. One way of doing it is say, okay. I'm going to put a quality control team. My governance team in the process of generating every data, and this is going to sit in the middle and verify at some points in that life cycle of the day. Data its beta acceptable to be shared. That's where system taking comes
to play. That system is going to have a massive bottlenecks and it's not going to scale. So how can we achieve a defined level of quality without creating just necessarily just controls? We need the consensus or definition around what constitutes quality as in what attributes to reuse to describe the quality of data. Is it complete? This is of Integrity is a timeliness, like, is all of the above and others So let's define those and in that definition is have subject. Matter experts.
This have domain people who actually know their data and how they can articulate quality involved in defining that and once that's defined, that's automated you. Let's put it into the platform as a plan for capability. The moment, you are instantiating it in a product, you will get out of the box, a library, or some SDK or something. That gives you the ability to now, calculate capture and share.
This quality metrics and then you will have observability that runs across this measure across, all of his data, paradise, and capture, those information shares that information also validates, whether you are meeting the requirements of the quality that you've done. So that's it.
Computational part and the embedded part that I haven't put in the title, is that this enforcing quality and measuring quality becomes an embedded concern in every single day product, it's not something that smeared over And added later on, it's actually from ground up built in the data, product itself is embedded in there, so hopefully that gives a good example of achieving I guess. Well, oh and and cohesive mesh of interconnected data products through embedding policies.
A standard policies in an automated fashion in everydays product and it have the teams that are responsible for guaranteeing. Those policies involved in this defining what these policies are. Yeah, if we can borrow things like from the application development rights, we have this concept as well policy as code. There are some tools in kubernetes clusters where you can embed this kind of policy. So before you apply something, you will check towards the policy.
And if it's doesn't comply even reject. So data governance is probably one of the least sexy part of the data management because they are things like pii data security and maybe should not be leaked out and it should not be exposed maybe things like data quality, right?
How much lagging for example the data Be and I think all this definitely needs to be governed because otherwise it's really difficult and you mention about observability and you use the concept from SRE where you have also data SLO. So maybe if you can touch a little bit about this data SLO? Yeah absolutely. So data products for people to trust your data you need to share a set of real time.
I was or at least as real time as your data in these information to give people trust that this is a Suitable data. So, again, the dimensions of that, I think I unpacked it in the book and probably don't remember all of them are top of my head. But the dimensions of that all around quality, there around timeliness there around completeness. There's a whole set of often in the language of potato people. These collect metadata is language, I don't like and I totally use because it's just a
casual bag of all things. But there are classes of really information additional data that you've got to provide for what purpose. So that the people that want to directly self-serve, use the product, they can self assess if this data suits their use case or not as an example, the distribution of their data. So if I'm doing an analysis, where trading a machine learning model for particular, use case up, perhaps like to have a very I don't know.
Nice bell curve distribution of the data and the samples that I can get for training that machine learning model, rather than by estate us. So how by is the data is. So these are again SLO is in the active or more about part up time and response time down tournaments so on and then the data world to the competition of part of it still has those concerns. But the data part of it, it has a different set of concerns that defies of usability, Mitch weeks
of a data. So thanks so much for explaining all this. It seems like a very crash course of data mesh. I hope people do study about this data may be from reading your All your articles are watching some of your talks. I think it's really an eye-opening for those people who work with traditional data management. So thank you again for this. I have one last question before I let you go. So normally, I ask these three things called three technical
leadership wisdom. Maybe if you can share some of your wisdom for us, maybe to learn from your journey from your experience, or your expertise. It's a hard woman. So maybe just a few things that I didn't do as well. They can share or things that maybe I did. Okay, I separate leadership from management. I'm a terrible manager, and you don't want me as a manager, but maybe I'm going to K be there because I believe in the mission, I'm a very Mission
oriented person. So, as a leader, you need to believe in your mission and create a mission oriented team and organization and continuously through communication, through reinforcement of like embodying, the bright Behavior to achieve that mission reinforce that and remind your teams and keep real. Lining the team, maybe there are different styles of leadership but that mission area to Visionary into leadership, resonates with me. I love working with people like that.
And then to get to that mission, you have two ways of going there. You have no way of leaving. May be some casualties behind like going in a way that not everyone can catch up and if you wouldn't take some soldiers along the way, but you want to make sure that everybody's alive you need to seek about every single member of the team, their
needs their pay. Pays their specific hopes and really it's about not only having the vision and charging the past but also making sure everyone can come along and
think Beyond yourself. That's probably an area that I need most help with personally myself because my mission oriented kind of leadership, usually has casualties and the way your people need to be able to trust you and believe in you, you need to be very self-aware in terms of your strengths and your weaknesses where you want to delegate. And where you want to actually take something on.
And if you're a technical leader, I personally respect technical leaders that still stay close to their craft. They still keep up to date with their craft as we all know, to technology moves really fast. So you have to find a way to keep yourself relevant and up-to-date. And sometimes that means going really deep for a moment in
time. Get your hands dirty and coming back up. And of course, As your scope of leadership, Rose, your ability to go really deep diminishes because there's time doesn't allow for that. So, having carving out space to go deep when it's needed, even for a very short period of time. I've seen some technical leaders do that, I admire people who can strike a balance between the depth and kind of the breadth of knowledge and relevance of their knowledge. Well, really beautiful.
Thanks for sharing that. I think it speaks to some of the leaders where they are more efficient driven as well rather than managing Well, so thanks for sharing that. So, maybe Zama for people to learn more from you maybe about data mesh or just to reach out and follow up with the discussion piece, their place, where they can reach up, well, as it's now really Twitter and Linkedin will be the place.
So I listened to both channels but hopefully soon my company's website will be up and we will have jobs and have places for people to reach out directly through that, but that's not up yet. But when it is, I will let you know and you can share with your network. Really excited to hear about that. So many different data mesh Technologies maybe will be coming from that. So thanks so much for your time, really a pleasure to have this discussion with you.
It was wonderful to be here and we thank you. Thank you for listening to this episode and for staying, right until the end if you highly enjoyed it. I would appreciate if you share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you are new to the podcast, make Subscribe and leave me your valuable review and feedback. It helps me a lot in order to
grow this podcast better. You can also find the full show notes of this conversation on the episode page, at Tech Legion, o.f website, including the full transcript, interesting quotes, and links to the resources mention from the conversation. And lastly, make sure to subscribe to the show's mailing list on package. You know, dot f to get notified for any future episodes. Stay tuned for the next technology. No episode. And until then, goodbye.