3. Make BI Great Again

Now, the role of be either for us evolve into not just providing data to essentially what I call a identifying the the needle in the haystack, right? When you're given a data science is part of an engineering team or as part of a an end to end product, where you are model is embedded. You're solving for actions, whereas it is in bi or solid footballs gravity, and I also keep encouraging the bi developers in my team. They don't call yourself a microfiber Jeepers.

Don't call yourself a cab. Don't call yourself about here. We have our. So you are essentially a bi. Hello and welcome to data. Shatter the podcast on all things data. This podcast is a series of conversations with experts and Industry leaders data. And each week. We aim to unpack a different compartment of the data suitcase. I am your host, Karthik chassis that I want blogger newspaper, columnist book, author, and a former data Etc conceited.

I currently head analytics and business intelligence for delivery. One of India's largest logistics companies. You can follow me on Twitter at Karthik s that is Kar Phi. K s and read my blog at no Intruder.com. That is n. 0e n t hu be a.com. All opinions expressed in this podcast, belong to eat and my podcast this and I do not reflect the views of any organizations. We might be Associated, nothing discussing his podcast, should be taken as Financial or legal address. I would be lying.

If I were to say that business intelligence does not have a branding problem. What if period of time, the term has evolved to mean, a very specific kind of software engineering, which involves connecting database. Visualization tools. However, business intelligence is not necessarily limited to that my teammate delivery. For example, does it fair amount of statistics and over and modeling and machine learning and make the odd report? And we are called the business intelligence team.

And originally if you think about it, business intelligence was all about making business decisions through the intelligent use of data. So my guest today also works in business. Intelligence is name is Balaji Cooper Swami. And he's director of business, intelligence products and data infrastructure at YouTube. He's had a long career in data and analytics and decision. Support having worked at Captain one McKinsey City Mo Rocca of Commerce and lows. So you're the director of

bi-products and data infested. What exactly does it mean? What do you do? Yeah. So, one of my core responsibilities is to kind of look at in the diverse sources of data that there is and connect it to the needs of the business and needs for decision-making that various stakeholders have beer in terms of partner management of weed. In terms of advertisers

management. My job is to connect the data to actions that The they want to take based on the based on information that they have to ask about.

Generic question. What exactly do you mean by bi? Because I mean just if you think about it, all the terms related to data, if you think about it, it means different things to different people. You say data sends as 10 people about data science. Until you did ten different sure. Yeah. So what is your definition of business intelligence? Yeah, so when I think of business intelligence, So there is a saying we call right data's, worth pennies and

actions are good dollars. So, I believe that the role of business intelligence in an organization is to convert those pennies into Dollars, which means that there's a ton of data, especially when you're in a business, which is digitally focused. There's just so much of user data being created, so much of operational data that is being created. How do you kind of take that and an influence the decisions and actions?

Since that the that variety of leaders, want to take across the organization V8 starting from say, an analyst where I see to a manager. We could be a product manager with an icy. You're all the way up. We'll see. So that I believe the job of the bi is to make their lives Easier by providing insights and actions, that matter, to them based on how they can to solve for their. Okay. Ours and to solve for their results. Okay, and what do you mean, you think the market?

Things VA does this tool? So my I think that doesn't mean bi is relatively old term. I maybe there's a probably need to reinvent the terminology of what you call BI, but but be I started off and like, older days where sap and Oracle it basically meant data warehousing and Reporting, right? Okay. Yeah, and that is what generally people think bi is essentially a job of just collecting all the information and The spitting it out.

Whereas, whereas I think with the with number one, just the volume of data exploding it. It would have been fine. If you had like five metrics that you want to look at. And that's all you needed to do. But whether demand for data on the hunger for data increasing across across the across the organization, as well as the volume of data, also continuously increasing. Doing the same thing, essentially is becomes very pointless exercise because there is too much data to do any analysis.

So, Now the role of B A. Therefore has evolved into not just providing data to essentially what I call a identifying the the needle in the haystack, right? There's a big case type of data, so much. That aesthetic that how do you identify the needle? That matters to you? But how do you identify the most important aspects of the data sets that matter to you? So that you that can be? That can influence your decision-making on a daily basis. So that is how I believe it started off.

As a hey, I'm going to collect all of those information from a variety of sources into a data warehousing system, which is on opened some reports. Yep. To what I believe now is is turning that and using using algorithms and statistical models to find out what's actually going on in the inside the data. And we only surfacing data that matters only surfacing insights

that matters. Okay. Thanks. Okay, I think Our definition of bi is broadly sort of aligned with my definition of bi and what we in my company also believe that he is supposed to do which is like sort of help Drive data into actions, help help, help the business. With the sort of intelligence that can be derived from the data to put it in a very simple manner. However, I mean, what I find is

that I have a bit of herb marketing problem in. This is, like, recently. I was hiring, we very quickly agreed on the sell very quickly and things like that, but the guy was like, can you I mean something other than bi because in the market there is a bit of a Down Market, sort of a field to the word GI and this is something that I've heard from

other people as well. Not necessarily people have been hiring, but others have also mentioned that like, because bi has gotten sort of become synonymous with people who, like they'll ask, how have you worked on that Tableau or quickly knowing these tools knowing how to build the pipelines and things like that. So so I mean I find that The chicken wing ba has got a bit of a marketing problem. I don't know if you have faced that as well. Yes, and yes, and no. No, I think it depends about it

depends. But thinking about how the vision for the, bi is usually pitched to. And I think I really thought when you thought about bi again, right? The historical job, that'll be, I professional used to do this. Is essentially, just a data collecting Under The Dumping job, which may have, which is

exactly. As you said, right, like a take build all these data pipelines and put it in this, which I believe is immensely useful guidance cement Foundation, that is required to essentially do anything with the data you want to add. But what from a from. But I think where the the role formal rule for how it is evolving, now it is thinking about how to stink like a product manager, right? So I feel it is more. Or actually a product rule versus are versus a, just a bi room, right? So go.

So I also think that there are very specific skill sets. Rather. There are parting about three, four layers of skill sets that are required in order to be a well-rounded bi to form a well-rounded bi team, right? I don't think I can probably have a one person who does everything completely. But what I think, two things, Fit like a product is what is what will Elevate the marketing for BF. But I don't think people look at it as a product because the think it's still a data dump,

right? And that is historically, it's been associated with that fact, but when you start positioning yourself as a data products team, hey, where your job is to build scalable Data Solutions. I mean, think about it from a standpoint, a lot of software as a service business that has been set up and it really A lot of software service which have now a i animal tagged to the hair to their names.

In the end. They have a huge component of their work is bi. It's actually about collecting data from a variety of different. Maybe your client, maybe, third-party, consolidating them. And then starting to surface them in different ways or not in one form or the other day doing a roll of bi. So I think, I think the marketing problem exists, but I have found success in positioning, it more as you're

building. A product and out of the product is actually understanding the use cases and then connecting the data back to that, right at the decision that use cases and connecting the data back to that. That is how I have found success. And but when you actually really look at when you split the word business intelligence, right? That is why I like it is, it is basically just artificial intelligence, but you're calling

it business intelligence. But yeah in the end of the day, that's what it is. So so I think there is a stigma associated with the term. Be II.

But I think we can use the, I think we can use the mahout where the market is moving with respect to actually building data products and thinking from a product mindset thinking from a design mindset and then bringing all the data to solve for those use cases to essentially, to essentially show to the candidate or shoe to your team members, how much of a broad range of skill sets that you can, you need to bring to the table. And also, you can pick up why?

Does rolled and that is probably the sets you up into a role in product manager. It sets you up into her role as a data scientist because you will be doing transporting, it will be doing anomaly detection that again as follows. Yep. It sets you up as a role of a user design because you'll be actually it's like understanding decision-making. In a business is one of the most

important aspects it out spit. Also it also gets you up to speed into the most important factors in a business, right? Like think about it. I always say that. One of my first jobs and I went no Capital One was to do monitoring. Okay, and, and within six months, I knew so much about the business at not even an expert in our particular field, New because I used to always sit with the executors. I used to understand how they

look at the business. And yeah and solving for that also gives you and bi also helps you kind of grow and as a leader. Because are you starting to look at the business from top-down also from Bottoms Up, of course, so I think it just, it just gives you a broad range of skill sets and obviously the tech skills associated with staying up-to-date with Streaming data that's data and then injesting that. So it has a full gamut of literally what you need to know to be an analytic professional.

You can Branch off into wherever you want to go after that. Yep. I see. So I think is a good time to for me to kind of read out one of my favorite quotes. I think I have, I have a

newsletter on data science which are where I probably use this code three times already. So this one is it's by this guy called Kay Kyser Fung. He is a Kaiser. Adjunct professor at NYU. I like Columbia or NYU. So on this block he had written this a couple of years back. 'Hey, I'm quoting here. The data science Community is guilty of talking down on the business intelligence function. There's a misperception that bi is for Less skilled people doing

boring things. The reality is, there is more science in bi than, in so-called data science, science of, after all is about figuring out. Why things are as they are Engineers by contrast, use our understanding of science. He's debating touch. Yes, and I think that very, very relevant to where be, I was thought to be a bias is evolving or where their vision for being needs to be exciting.

It's very relevant. Yeah. It's like because you need to kind of like if you think about it, like it's almost like somebody will be like you could be building a simple sales dashboard inside. And somebody could be the question that will be asked, is why did sales go up this month? Let's say, for example, it's a good problem to have. But why did the sales goal you need? Need to be able to intelligently dig in and the better you can present your information right

in the dashboard. And so on rather than let's say I mean in some cases I've seen like especially when you have companies that treat bi as a function to be staffed by people less skilled people. What, what do you end up happening? Is there's a constant back and forth between the business and the bapi saying that be Isis. Sales went up business, has why did sales go up? Then the be Isis or sales. Went up in California? Why did? Is the go up in Calico?

So this is back and forth. So the I think in some sense is one job, one job of the bi to sort of pre-empting solve this problem. Right, anticipate? What questions are going to come and then like present the data in a more intelligent Manner. And as you said, like, sort of pick up the needles from the Haystacks in sort of present, what is actually required for the manager, rather than, like, just just dumping the whole thing, because that how it was

meant to be done. Yeah, and a lot of times I have seen demos of dashboards, even from vendors. They come unto me o hero. Is a top sales and then they give you an inability to drill down drill down, drill down, drill down and I keep asking them the question, but why do I have to do that? Right? Yeah, like when when I know that the data when I know that the gold is lying underneath. Why don't you can why can't you just bring up the gold upfront to me?

And then why do I actually have to do so that is really the role

of Pi? So what's your what's your view on interactive interactive dashboards ended? So what do you mean by interactive do interactive dashboards again, like something like, I mean like where you as a so. So again the way I look at it like dashboards can either be like static or interactive. Static is like I say this is. This is how things are, and I control the message and you just consuming interactive verdict.

Okay, if you take it for yourself and like you can do the cuts, you can do your drill Downs, whatever. And you figure out what you want to do. So, what's your yes? Yeah. Yeah. Yeah. No, no. God encourage. So. So I think the road Of bi is very, this is where it is different from data science,

right? Where you think about data science and embedded data science, the the their majority of the time the decisions are already made by the algorithms are made by the models and you are essentially living with the the cost of miscalculation versatile model does, right. So that's part of the life. As you're automating that decision, yet. The role of business intelligence do is very different from the standpoint that there is more or less. Usually a human layer.

For the decision is actually made figure. And, and to understand that human layer and understand. How do you solve for that this? How do you solve for? The right decisions? Made by the human layer is very, very important as part of when you think about the entire bi-product. Yep. Why I say that, bringing it to the context of the two things that you mentioned writer. Static forces interactive now now depending upon the user, depending upon the person who is

looking at the dashboard. There are variety of You must be said that they may want to solve for and one of that could be. Hey, I want to know the what happened. There are a few folks who are like, hey were probably Understand the models to understand the data science is

going behind a day. Say hey, I trust the information and then I need to move, but there are a few folks who will be come back and say, hey maybe that job, that pnl there is their bonuses dependent on it. I cannot take a decision that your model is putting out me or the what you are showing me as a as a god-sent answer. I have to understand why that is there because at the end of the day, it is why my neck is on the line for the decisions that I'm making.

Yes, I believe that is an art of showing the recommendations first, but then explaining why is it such a recommendation happened to an interactive manner. Is what I think our dashboard should solve right interest rate. It should not be a less of. I will give you the word and you can explore it. No, that is not what it should be. You have to neither. Here is the recommendation live or die by it. The second. I think the answer is little bit between which is to say.

Here are the recommendations and then you should immediately, and then ate and ate It and it should be interactive enough to take you through. Why such a recommendation was made and what is that slice of the data? Because finally, there is a raw data Trend that you have captured in the business intelligence that is subsurface top. And also it is great to actually have a human layer of validation of that to see whether your models are acting picking up, the right stuff for Mark.

So I think they're, I think the answer is a little bit both, but instead of interactive from Top down and started from the below. So I believe there's a, there's a, there's a marriage that can be made to both of us. Good. So, okay. So now we have grabbed the kind of father. I think you've got the you mean comparing bi to data science. You've been sort of talking about how it's slightly different from data science and so on. So I mean and I think you and I

sort of believe that there is some amount of data science that goes into bi. But I think the unfortunately, the ladder Market income statements, that doesn't really believe that because it is it's about to order the. So if fix it, if you are a data scientist working in deai, how does your job apart from accounting for that human element? The that's there in the decision-making. How does your job sort of vary from being a data scientist on the, on the product side?

Where you're sort of, like, where your output is going into the tech product? And things like that? Yeah. Yeah. Now this is very different. I think the very different from standpoint that when you are, when you are given a data science as part of an engine. Team or as part of a, an end to end product, where your model is embedded. You're solving for actions. Whereas it bi or solvent for causality, right? So there are two different kind

of problems that you're solving. And there is really I mean, in fact if you look at it as you said, but of the the big motor design, I'm going back. What the professor said is really understanding causality, right? I think for us as a business intelligence when I think about it and if you were Ready. Dissenters, right? I think there are. I see there are three layers, right? One is getting the raw data. Is generally done by your date. Somebody who's talented as a

data engineer. Alexander is a sec. The second layer is what I call the cap. The metrics. Calculated metal clear. This is basically you're aligning on cleaning up the information and making sure that the data and the metrics for calculating sales, you know that your poxy optimism for the right sales number and so on. Right? Yeah, and then the last next layer is actually, I think is a model modeling layer, the modeling layer there. Is is about Trend detection, which means that you've got

understand? How to detect Trends, right? Because you need to have a benchmark need to understand where you're comparing that. Well, how are you? What you're comparing it again? So now there could be a variety of different benchmarks, right? And especially in an environment that we are in. You cannot use your ear over here as a as a benchmark. Just live with it right out yet.

So you have to think about ways in which, you can be smarter ways in which you can actually use machine learning to identify. What, what should have been expected and what is the trend? And, and whether or not The actuals are along those Trends and and there are also ways in which you can triangulate through through rules based algorithms as well. Right?

Yeah, in terms of triangulating through year-over-year, quarter-over-quarter benchmarks, percentile, benchmarks, and there are so many ways in which you can calculate Trend. Don't thinking about, how do we do that? And calculate the trend itself is one of the big on blocks that you can give to the business that that they don't have to do it manually. Yeah, then there is an aspect of

as you said, right figure. An hour depending upon the use case that there could be a normal use that you want to solve for because there is always something that is broken, be it in terms of in terms of credits. It would be it would be a default rates. There's usually some kind of a classification model angle that comes into it in almost every aspect of business where you have to think not only from from a linear Trend standpoint where

you have to catch those catch. Those escalations or cats. Something that is broken or in the retail World, especially the pandemic. We all know that the supply chain went through a huge Crunch and and orders were not being able to fulfill and they were getting cancelled. So solving for cancellation. There are different ways in which you can apply the models to say, how can I prevent something from happening one? So, let me take a step back. Right? So one is to say, Hey you cute

is what happened? Yep. Second is the cutest. What happened, and what? I say words of what behind a wall, which is yes, a third needle, the segment, the most important segments, that move that method really is your just one second. Yeah. Sorry, it's my alarm going off. So just going back and saying, hey, there are three aspects of it to four aspects of it. First is understanding.

What, what is the data? The second is understanding the segment of the data that really matters that you need to look at. Yeah, and then the third is what are the cause of drivers for that segment. So that is when you think about actually good actually thinking about not just going drilling down what correlating that with factors that that you can move. Needles that you can move, right? And then the fourth is, okay. How can we get ahead of the problem, right?

Rather than now, even till now we have been detected. No problem. Yeah, it's job is to how do we get ahead of the problem and solve the problem before it actually happens. So, so when you think about this, Literally, after solving for the data, almost everything that I explained is generally a data scientist job. And especially when you're thinking at scale, it is not a it is not a hundred line axle that you're working on.

You're probably working on gigabytes and terabytes of data with probably thousands into thousands of Collins to look at. Yep. So from that standpoint, it cannot be done by with pivot tables or anything else. It's got to be solid at scale. It's got to be solved through through Machinery. Yeah. Okay. So coming to machinery. So, I mean let's just go a little deeper here, right?

Like so when you are building a machine that, again, the question is similar in terms of like, if you're solving the data science problem that the, you're solving for Action, which goes into a product versus if you are solving for a sort of a bi problem, where the where you're solving for to find out causality and things like that. How do you, how do they ml

models themselves change? Is there a certain kind of models that that are favored for the data science kind of a thing and certain kind that you would favor for the I mean in terms of large classes and so on like that you favor for the bi K Pi world and so on. Yeah, I mean it's a very interesting thing to think about, right? Because when you are very when you have an and this is how I

think about it, right? And again I go back goes back to why are we building the model for each of those use cases, right? Yeah. Building a model in a product to actually make a decision. So so not to to actually implement the decision that the Already been, right. So so from that standpoint you have to, you really have to optimize for, you have to optimize for the right decision. Right? Which is BS that verses explain ability.

When I think about how much you have to explain a model versus how right a decision has to be. Yeah, you solve for the decision being right versus experiment in bi while. Many times. I have seen that we end up, we end up doing classification. Models. And we end up trying to figure this piece out. And we and really a NN model comes out as a, as a as the best model yet. What we end up going with the logistic regression though. It is less slightly less

performant. I can actually easily explain the factors that is going behind those logistic regression to the use to the users of the tool and and therefore can explain to them what happened and that enables The role of bi, which is to actually power and decision versus actually just implement, or just do the decision. Right?

So, I think that is where I see, as a big class of difference with respect to how a model, how you should think about a model in a product setting, versus how you should think about a model from a, from a cause of a risk, especially aspect, you know, trying to explain setting. That is 1.

Then the second piece also is that when you again, just because of the fact that there is a human error associated with that, you got to be very careful with respect to the input drivers that are going in as well. Right? Yeah, not be in a several times in a product said embedded model has seen like people just dump a huge set of feature set. And then then the model.

Do its job right here. You've got a be in a situation where you actually have to be very careful in thinking about what are the factors that opinion? Because in the end of the day you're gonna surface that you have to see whether it makes business sense.

So I think there is a, there's a, there's a you go to A little bit more from a standpoint of actually put yourself more and empathize with the business user in a bi V than you have to in a world where you're actually building a data science model

for in a product s. So the oak is likely cycling back sake. So you seem to have a bit of an unusual profile for somebody who's working in bi and so on. So, how did you get it to be? I ache. What, what you? What was your journey? Like, I know, you were in Capital One and Mackenzie and so on but what is your journey like?

Ya know, I think the interest Eating crediting curve, as I said, initially started off at Capital One, a thing as I told you one of my first jobs was to essentially own portfolio monitoring. Right? And I and I and I saw the power that it had in terms of just teaching your business and and and almost always I are a realized that the whenever I could went into new business and build up seeking a role like that because it immediately taught me the business very quickly, right?

So so I understood the power of And I went into good and then and even abstract adults were on the time and when I went into Mackenzie as well, I started this, that's when you remember Tableau and all of those things are actually coming out because all the monitoring I did before was pulling in from from teradata and then, and then writing me be scripts to automate it in SF in Excel and all of that stuff, right? And yeah. And all that's what I used to do

in the past. But then you started seeing the tools that will coming in like Tableau and other solutions that were actually the surfacing of. And I built a lot of in fact, even in the neck. And they did a lot of projects where, where where I essentially help think about understanding say, Call Center Performance and things like that through these trees through these realization to.

So while believe in never Left the concept of doing the bi work, but at the same time, it was about how do we. But I, but it in this inner-city, the first phase of my life. It was about just learning the technical skills. Improving. Hey, getting the data together dumping it and figuring out the trends and so on. Yeah, the second phase of it mackynzie was about usually I was is the work I did was part

of her strategy product. So it is about how do I how does that connect with strategy right corner? Center was an example. Other one was an example where there was literally, no, the client did not even have a mechanism in which they could store an information. So we literally had people track an order and write it down manually saying, what's actually going on and then we fed the data it would Excel and then we built out a dashboard to identify bottlenecks in their processes, right?

So varieties of different ways in which we solve for b, i from a standpoint, but we realized it was a banker secret strategy. Then the Phase of my life was at Boomerang Commerce, where it became where I realize the importance of scale because I that point I had to think about pricing decision for the entire world of retailers, right? How do we think about solving

the problem at scale? So so going from Bonnet, going from understanding the data science of the bi world, to understanding the Strategic importance of the bi word at McKinsey, to understand in the technique importance of technology and platforms of the mechanic at boomerang. Commerce is kind of what join all the pieces of the puzzle together in terms of me thinking about continuing in the Journey of a be in continuing the journey in the sphere of Pi.

Yeah, then lows. And right now at Google I'm kind of putting together the lessons that I have learned throughout my career and kind of joining the technology, the data signs as well as the business decision making to to essentially bring the insights and actions forward versus say, just just Bir technology can do by themselves, right? It's kind of putting them together and getting a hold of getting a sum that is greater than the sum, of course. So I guess you're sort of some sort of.

You've done some data science, work in the past and you've done some strategy work in the past. You've done some dashboard work in the past. So it's almost like all of this has sort of come together for you to kind of. Now, sort of do your bi work to put it. Yes. Okay. Awesome. Awesome. Awesome. Awesome. So, I think in some sense, in that sense, It's all about bi in play. Yeah, I think you described it. Right? Like it's up.

It's a sort of a product product manager, kind of a thing gets on. So now, like, I mean, slightly

switching tracks, like look for a lot of people when they think of bi. They think about it in terms of tools. So, especially in the market as it stands right now. If you look at the bi job market and the other day, I had joined the radicand EI and Reddit subreddit on pi. And there, it was all about people were discussing about It is Tableau better than power bi better than it is all about tools. And it was about the knowledge of to send someone to question for. You, is like your Dubai.

I mean, I know you work for Google now, so you kind of might be, you might be considered to use mostly Google's tools. But like, are you a sort of, like, have you been sort of constrained by to sorry. Do you have a favorite tool? Or like, what is your or in terms of? How do you how do you look at this entire bi tools universe? It's a, it's an interesting question because I think it can its, there's no, there's no straightforward answer to that,

right? When you're solving for, when you're solving for a success inside an organization, right? You've got to use the tool. So thinking about and I have done this test at variety of different places. Bringing a new to bringing a favorite tool. M. Versus. Working with a tool that may not have, all the features that you have but has the ecosystem that supports it.

Yep. Right. And I have found always that the tool where the equal there is an already, an established ecosystem that supports it almost far out performs in terms of the, in terms of the impact, overall value created by the thing, don't do it may not be the best visual tool, though. It may not give as much, you

expects ability. But but the fact is that, if there is not Of a platform or a set of ecosystem that is supporting the, a particular tool you will end up becoming you'll end up basically having something that could shine. But there is no, there's no Breeze kind of making it Shine, right? Whereas if somebody even if something that may not be a shiny, but if there's an entire

team that is lifting at all. I believe that drives Home Project having said that, having said that I have, I have work. In technically, I've had now had the opportunity to work and all three different areas right at the start of that. I work for, we were an aw shop, right? And then, so we've had an opportunity to work with work with red shift, and then Tableau. And then in between we did had have a snowflake as well as a

computational platform. So, I think that really really worked well with a We have low steam was embracing and open source, open source Technologies as much as possible. So, that is an area where I got an opportunity to work with a variety of different open source, tools, like like Druid and, and Kyle, and so on. Then, then the next piece is now, now in Google. Now, I am in a Google shop. Make, oh, yeah. So from here, I'm getting, I'm getting a sense to work with. Not only tools.

That are available in the market, but also a lot of in build tools that are available for for googlers. And yep, and I'm able to get to see a lot of the wide variety of it. And honestly, this is where I write my opinions are far from the standpoint that there is actually no best to it's about tools that the tools that that you are. And I also keep encouraging the bi developers and my team to say hey don't call yourself a microfiber is a person. Don't call yourself.

The photographer said, don't call yourself a power bi person you have our. So you are essentially a bi person that you've got to be in a position like similar to a software engineer. They don't go out and stay. Okay, if a C++, I men if it's not C++ I'm out and you cannot

be in the same word as as that. So you have to you have to essentially think as a bi professional and then tools are just tools are just the tools, right? But you are the one who create in the value. So a Again, tying it, behave in. I think you are in my definition of bi is that there's a lot of intelligence.

There's a lot of data science that goes into the into the computations before let's say the final dashboard or equal to whatever is developed and so on. So in general how compatible are the sort of the tools out there in the market to sort of building in this kind of intelligence. I mean, I have a confession that I have really despite having been in sort of broadly, this business for about 10 years.

Now. I have not really taken to any of these tools and I sort of Do everything from first principles using our or something like that? But so how how compatible are there tools to connect? Because my impression of some of them is that they are just. You just need to write the pipelines to go from the database to the thing. I'd like, putting in the intelligence is, is, I mean, it's very nearly impossible, in my opinion to put in intelligence into an SQL query, some sense. So, yeah.

No, I think there are several, there are several ways in which, I have seen to use these tools and insert intelligence. So one obviously is to have your pre calculated data sets. We are feeding into your Tableau, right, which means you've already done, your models, your scores have been calculated and if that is the data that was feeding, right?

So so if you can create a separation of data set and secondly and I believe there are rules are coming up and I think Booker and Tableau they're creating ways in which you can actually do detection prediction and a Addiction and even simulation scenarios inside the inside, the pool itself so that I think is a so, so the the the software's are essentially making it easier day by day to do to use the tools that they

have. But again, I think about it for somebody who wants to start from first principles. It may not give you as much flexibility in terms of the modeling techniques that you may want to use. It may not give you the flexibility in terms of in terms of precisely the choosing your thresholds and so on as you may want, but the tools right now are actually capable of giving you very lightweight ml, in terms of just inserting a particular algorithm and just finding out an auto a score.

Right? There are the there are tools that are helping you do that. Even if that Tableau even inside looker and and I'm assuming property is well, but irrespective of that. I think the best mechanism Best way to optimize it is actually pre-calculated very similar to what you would do, like use our or Python and run. All your models in in your computational platform. And then use the visualization tool mostly as a user Discovery.

You user user experience problem versus the computational platform. But if somebody who's not as experienced as say, you are an experienced data, scientists are in terms of building that I believe there are options. There is options for them to do something like, wait, inside the tourists, were somewhere in the beginning.

I think you had mentioned that like, he look at bi as a product where the you have people with different sort of skills coming together and so on. So let's say somebody is looking to develop is looking to build a new bi team in some organizations. Let's assume that for take my situation where I I joined six months back in until then we didn't really have a bi function and I have been setting it up and so on. What's your view on integers?

How do you build a deep? What is sort of skill sets to look for among the different people. I am assuming that you need to bring in people from different sort of backgrounds here because it's not it's not like you can't bring in let's say 10 DIY professionals to put a team and so on. I guess you need different skill set. So what are the skill sets that you would sort of look for when building a team player.

That's a great question. So so I think again just going back to their appeal when you think about the layers from data. A Shinto. Action generation or inside generation, right? So I think there is one one team that is required to own the data and the logic, okay? As 11 basic. I think there are four layers spheres of ownership. One team owns data across or or one skill, set the bones data and of it is not dripping or call it a team.

There's one skill set that bones insights at scale which means that they are responsible for understanding the power inside. Generated in the business and also producing them at scale using data science. There is one team that owns consumption and user Journey rap. So say. Hey, now this data is there. How do you kind of translate all of that into a into a user journey in terms of how whether they want to consume it as a dashboard that they want to consume it as an email?

There. Is it an alert? Or is it like just a written summary of your insights? Whatever it is? That how do you want to consume it mean? So that is for third? And then the foreplay, which is the canvas that is spreading across as what I call the product management new. Yep. They open both the users success as well as team success. Yeah, so so I think these are the four buckets of talent pool. The needs to be brought in to solve for a b are. Now if you're starting from

scratch, what is p? 0 data is P0 that stakeholder management is p 0, which is hey, somebody who kind of understands the use case, understand solves a problem. I think that becomes P0 so you've got to start with that, right? Which probably, Many of the companies already have invested a lot of time and effort and at least with the data over the course of the tower over course of the years, but but I think if you're starting from scratch, that would be the first thing you start.

Then you, then you would think about adding in bringing in the data scientists to as part of your scrum teams to think about. Hey, how do you now look at now? The data is being used. How do I get one? How are they looking? However, the business stakeholders looking at, at what? What, what kind of insights are they looking for? How are the Actions influence.

How are these houses data influence into action, so that you can now stream line, the process and favorite inside the scale, and the third, and I user experience is very important. Again, if it is an external facing tool, that would probably also become as p 0. But if you have an internal bi setting us up for doing an internal decision support, we can take the luxury of giving them a not perfect you X so that so that we can continue to build on work on it.

As a P2 and if you are thinking from a standpoint as a bi leader, but this is what we, this is how I would think of P0 P1 and P2, in terms of how I would set up the set of the organization and bit of the organization while also creating value in the same same page. Yeah, and what kind of skills would you look for any time in the, for the data? And logic? You'd is it like data Engineers or what kind of skills. Are you looking for?

Each a buckets? Yeah. So let's go into the data piece right now. Now, as I told you a lot of it is dependent. Pending upon the platform on which you are sitting on, right? But so so several times. I have seen, you actually need hardcore data Engineers or if if that entire engineering combine is made, it is become very easy through some kind of a platform initiative. You could go with. I mean, I've seen technical analyst also grip, be pretty good at managing a chronic datasets, right?

Because there are right now we have tools which are which we just help you subscribe to streaming batches of data at Easily. It helps you build your own QA alert. It helps you were on your own car, bomb data, consistency checks, without doing anything, all all by, just all with SQL. You don't have to do anything. You don't need to know any other language in order to all of this stuff. Yeah. So so with that said, there is a team.

Now if the tools are not available, you will end up needing somebody who is who is much more well versed with working with data platforms versus a somewhere. All of that is made easy through the platform. The second piece is as you said statistical analysis. So, this is where I mean, this would be your traditional data scientist data scientist. Rules folks, who have again. I would look again, my thing here, as you said, right?

It's different from maybe a product data, scientist role, who may be well versus an engineer. The, the team, the person here the has to have empathy for the business, right? And from that standpoint. So I again, I see a lot of analysts who have behaved in the past who have experienced in our who have actually probably got under a master's in data science or analytics seem to be very good candidates for this particular road because they're able to do business analysis. Very good.

Well, and also able to nothing, how do I solve that business analysis at scale using using algorithms efficiently, so that would be The second please. So user research. This I will admit is one of my areas where I'm still continuing to develop and forming my thoughts and opinions on. But I think you did read a statistic that most of them are companies between or in 2020. The indeed. There's they're engineered for ux design.

Ratio was about 20 is to 1, whereas now it's about eight is 21. So so, so the focus on user journey is obviously a user. Ux design will obviously increase. Every day. And, and I am constantly thinking about how how do I build in that? How do I bring in the right? Right. And I think the talent is actually baked is this missing, especially in the bi were, because again, you have somebody who had who understands the data, as well as the business to kind of Be Sedated in the

center. So from that perspective, and I think somebody who's interested in solving for this has a, is, will probably be in high demand and And, and I'm really looking for folks to work with folks who are in that area.

And then the last piece is your product managers and inexperience analytical leaders do really well in this because they also read because you need to kind of span all the way from data engineering, understanding the metrics of the logic, to begin sighs, working with the stakeholders understanding understanding, how the CEO makes a decision, which is how a director makes a decision and kind of Geling the all the dots

together. So I have seen several senior managers of analytics, or managers of analytics with with good, with good, high ownership of high ownership and accountability. Right. But they really become good product managers because they also know how to lead some time weight and your lead without Authority as a product manager. And and also they have probably done many of these jobs that In one form or the other. And I think. But I like, the clear weakness.

I see in this entire funnel in the talent pool that I see outside is at the ux. Yep. Yep, because I think ux you need to connect you to have an appreciation for data as well. They did, in this case, itís, not a general. You've exigent aux engineer. You need to actually have somebody who who can also do analysis. So it's like, if there are be as water, bu X, that would be amazing, but you don't see many people switching from vs. USA. One of them be to data science of great engineering.

Yep. We are actually I sort of got lucky that I got a former front-end engineer who wanted to move to analytics. Exactly right as you. But why don't sort of? So one other comment that you sort of made here and there, just like you kept talking about

scrum teams when it when it came to be. I so scrum. I mean, I don't exactly know what it means, but I know that it comes from the agile, methodology and things like that. So what's your take on the usage of agile in bi? And like, I'm assuming you have some experience in that Lake? The pros and cons of that. Yeah. so, So, I firmly believe that. The benefit that I get some modules, predict abilities, okay.

All right, because everybody knows what's happening every week, this predictability and transparency. Yeah, right? And I think, yeah, I mean you, so from a standpoint, everybody knows. So if you think about what is the big detractors of success for a bi team one? As you keep getting Pink by all the users saying. When is this going to come at us that cannot calm? Well, I need this. I need that, right. So there is that whole list of

priority issue. The second the second issue comes where, when all of this happens you have to, you actually have to disturb the team who are actually building the solution in order to tell them. Hey Venice is going to come - are going to write. So that is one. This is the third and then and may never have seen when you not had an agile is. There's no clear roles and responsibilities. Like you sometimes have the D data engine, somebody who's killed, wants to be a data engineer.

Do we have you work? Whereas somebody who wants to do statistical analysis? Is is doing is just bending. I just had like some reports. Yeah, so so so now that you have secure lab clear roles and responsibilities means that you are able to actually put the right people in the right, get the right talent and and fit them in so that they feel happy about their job and to get job satisfaction. Yeah. Less disturbance in a, in a

standpoint, right? If you're not releasing for two bonds, people will always disturb you but if you're releasing every Two weeks people know exactly what's coming in at there. Is that level of transparency that comes in? So given that people don't get Disturbed, there is a little bit of a cost to that which is that there are scrum team scrum meeting that happen every day and we have several terminals that needs to do that.

We need to do there's a cost of it but it's a predictable coughs, it's not an unpredictable cost, right? Some suddenly. There is a CEO asking question. Then you got to drop everything and then you've got to solve for that and then everything goes Haywire, but here it's a predictable cost. You know, exactly the product manager is able to handle. All of that, chaos that can protect us from Team from, from, from the, from just focusing on

the work. Yep. Then the last the last piece under the last pieces, I Traditions obviously, right? I think in a world where we want to and I'll seem too good. Pros and cons of I trading fast versus making the product usable really invisible before you deliver it, and I think, I think that is very fascinating ideas

on both side. And I think it depends upon it depends upon organization of the culture of the users, as to, which the way you want to lean on, you want to hydrate very fast and really something very Scrappy, or do you want to really make the product usable before your release? So, so, so, that is so that is also an area where Deep house because you're able to get feedback that you can go back to, you don't have to wait, four months or two months to actually

get. So, so yeah, in order to release every two weeks, you need to actually have, you need to actually have operate like a factory, the cons of it. As I said, one. As you said, there is a cost associated with it and, and sometimes and I know Engineers hate attending meetings. So there is regular meeting that was scheduled to people. Keep saying, hey, why do I need to attend this? Because the sea is calm, most of the time, there is usually a stop there is sometimes a storm,

right? You're paying a cost during the when the sea is calm in order to prevent stop. So Gap. So when you actually don't, when you don't see the star, people always ask the question as to, why am I in the meeting yet? When you take away those meetings and the storm starts hitting then we like people get

let's get that right? So yes, I think that is the so usually I have seen pushback from engineering to obviously the second thing is that It also creates a lot of impact with respect to all the other teams are extremely chronograph occur. Right? Exactly. After the most common eating with all your engineers and all your product managers being in a room and solving for all of that wood style puts pressure in terms of K. Maybe somebody taking a night call or somebody's taking D

call. But if you actually have, you would be it. It qisas that pain. But apart from that, I have seen generally have seen the benefits. Outweigh the The costs associated with that and especially even as a leader. I get I get a level of transparency of how the scrum team is progressing, and whether they're working for towards ago,

he has. And if there is a, if there's a need that I need to enter me intervene to, to course-correct unable to do so before, like, Things are Slip too far, right? So I think that's, that's really the help that I get from a shell as well and whatever the data science part of it.

Like, I mean, if you're in model building and things like that, the what I would using my old favorite favor, What I would classify as they start to work where like, you need to sort of like where it's the development, doesn't happen in a linear fashion. But like example, is that you end up getting your insight into only how conducive is that to a child? Yeah, and I think there are when you have data science in the team, then there are essentially again, there are always two parts to it.

Right? As you said right, there is one gotta go to definitely invest, time into doing invest time into solving for the big one. This is generally during data science and also any region, a free ride where you are solving for a big technical Observer ready, right? Like that. You gotta solve for our you got, you have to invest time in your in your Sprint's to do that,

right? While at the same time, you got also think about what a quick and dirty way in which I can move the ball forward and not wait for the end of the world before we actually crafted best model. Yeah, if it means that you want to solve because many of the Tanners, you said, right? How can you make the research usable on a date? Mrs. Slater, that's really,

that's really the role. Where the product manager, the data scientist needs kind of sit together and say, hey, if you guys are doing because many times you end up with a lot of bivariate to multivariate analysis that seeing, if there's the rule, rule space way in, which I can start getting some value out of it. Yeah, for before you make, except the different ways in which we can try to try to use the research in and monetize that we searched every

throughout, the Sprint's versus just wait for the final result. Thank you for listening to data shatter. If you like this show, please leave a comment, share and subscribe to the podcast. You can find this podcast on Apple podcasts Spotify or wherever else you go to get. Podcasts. Once again, this is Karthik signing off. Thank you.