All data scientists and all analysts should spend more time in the business outside of the data sets, just in the actual business to see how it works. They should be shadowing their colleagues who are in charge of either entering the data or just doing business operations because then you have the context and then you understand the columns that you're seeing in the data. So just understanding that data generating process is really important.
Hey everyone, my name is Henry Surya Virawan and you're listening to the Technically Journal Podcast, the show where I'll be bringing you the greatest technical leaders, practitioners and thought leaders in the industry to discuss about their journey, ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our journal. Hello guys. Welcome to another episode of technicianal podcast.
Today I have David Aspart here. He is the author of soft any data analysis problem. So any data analysis, I'm sure today you know that the topic we are going to cover is about data analytics or maybe data science or whatever data problems that you're facing right now, right. So I think I hope today David will be able to give some insights how we can actually learn from his experience. So David also have a podcast, probably we will touch on a little bit about his podcast.
So welcome to the show, David. Hi Henry. It's great to be here. Right. David, in the beginning, I always love to ask my guests to maybe share a little bit more about yourself, right. So if you can mention any highlights or turning points that you think we all can learn from you. Sure. So I've changed careers a little bit a few times. I mean, it's always been in the tech space. I started off as a software
developer. My undergraduate degree was actually in video games programming because I thought that's what I want to do. I like video games, so I thought, well, obviously I'd love to make them. And it turns out it's really difficult. You have to program some really difficult things around, like graphics and there's actually a lot more maths involved. And so you know, that was, it was a very interesting degree to do. And the thing I definitely learned from it is that I really like coding.
That's something I I wasn't really exposed to before that degree. And so I became a software developer and I did that for a few years. I was really enjoying it writing sort of enterprise software as it sort of happens if you're in a a small team. I was also in charge of the reporting. Eventually that became one of my roles as well is that the sort of we didn't have a data team as such in the company.
And so I took on a lot of that responsibility and ended up delivering answers to internal customers that they had by pulling data from our database. And over time, I sort of started to prefer doing that part of the job because I found that I was closer to the value generating aspect of the business. I was closer to real business problems where software development in a lot of cases is a little bit devolved somehow
from the business. Like if you're a software developer, you don't necessarily have to understand how the business runs, right? You don't necessarily have to understand how does the business make profit? Who are the customers? How did the customers make profit? You know, what is the operating model? Those things are not that relevant to software developers a lot of the time. Whereas if you're a data person, I mean, you can't provide any value unless you know how the business works.
And that's something I learned in that role. And so I thought, Oh well, maybe I should make a career of working with data instead. So that was my, I guess that was my second pivot from games into software and then from software into data. And then I did a masters in data science because turns out it's called data science. That's what I found at the time. So that's something that I should do. I was promised it was the sexiest job of the 21st century and all that kind of stuff.
And so I thought, OK, I'll study that. And so I sort of did a master's degree and then transitioned into being a data scientist in industry. And I did that for a few years, learned some very important, very interesting things about the difference between data science education and data science in practice, which I'm sure we can talk about. And then again, the sort of undercurrent this whole time in my career changes was that I
always wanted to teach. Like, education is one of the things I'm really passionate about. And I I was trying to find various opportunities over the years. But it wasn't until I landed in the data world that I found my niche of data science and education. And when the pandemic hit, it was the part of it that was fortuitous for me.
It was that a lot of teaching. Well, all teaching became online and so I suddenly had all these teaching opportunities that meant I didn't have to leave the house. It just made it things logistically a lot easier. And so in late 2020 I quit my job and started teaching full time as a consultant. And so that's sort of what I do. These days I call myself a data generalist because I've done all these different things and I haven't really pigeon holed
myself into any particular role. Mostly these days I do educational work, so designing and delivering workshops, anything from half a day lectures to A10 week accelerators in data science and Python And things like that. And I'm really enjoying it because there's a a variety of clients to work with a variety of problems for the people are trying to solve that I can help with. That's where I've sort of landed. I mean I don't know if anybody can learn from that.
I mean what what I've learned is that just follow your interests like whatever has interested me. I I just put everything else down and just went towards that. And that's how I've sort of ended up in my like fourth job. Hey, thank you for being part of the Techno Journal community. This show wouldn't be the same without your ears, and you are the reason this show exists. If you're loving TLJ and want to see it keep on growing.
Consider becoming a patron at techledjournal dot dev Patron or buying me a coffee at techledjournal dot dev coffee. Every little bit helps field the research, editing, and sleepless nights that go into making this show the best it can be. Thanks for being the best listeners any podcast could ask for. And now let's get back to our episode. Thank you for sharing your story. I think it is very interesting,
right? I think many people started their computer science study because of the interest in gaming, right. So maybe people love playing games. I also did my computer graphics course back then. I think it was, yeah, difficult if you didn't get the math, I guess. So I think the career that you took probably is also quite common for some people, right? So they started by being a generalist software developer, but found into a specific area. Dive deep into that and become a specialist.
And you also host a podcast house Tech Data Science. So maybe tell us a little bit more about that. What can we learn from that podcast? Yeah, so that podcast. Grew from my first real data science job after finishing my degree and my Co host Sean was actually the guy who hired me. He was the hiring manager who hired me into that role at the time. And you know, very early on in that job I realized and Sean was the same. He sort of came from academia
into this kind of job. And we both quickly realized that what we thought the job was going to be is not at all what the job is like in reality. You know, a lot of data science education is focused on tools, techniques, algorithms. And so you get this picture that, OK, well, I'm going to be deriving formulas and doing all this complicated machine
learning at work. And then you go in and it turns out, you know, a lot of the job is navigating the complexities of an enterprise environment, working with so office politics and things like, oh, the data's not actually available and no one actually has any solid research questions. So we have to find those as well. And so we were very, very quickly hit on this difference
between education and reality. And so we we started having these conversations internally about OK, what are the things we've learned about how industry is different, What are the skills that people should actually be trained on or at least be warned about upfront so people have a better picture of
what the job looks like. And one day we just said, well we've had these conversations quite a lot and every time we went to a meet up, we would talk to a like minded people and have the same conversations and we thought well we might as well just put them on the Internet for other people to learn from. And so initially the podcast started off with the two of us booking a meeting room at work and taking our work Samsung phone, putting it on the table and just having a chat.
And then eventually we became a little more professional and got some proper tooling and proper microphones and things. But that's how it started. And so currently we're running a season where we're talking to educators in the data space, because I think at this point in time, getting the education of future analysts and future data scientists right is really important. So we've spoken to people who are Python trainers, but also people who are spreading the
idea of data literacy. You know, we've talked to a variety of people and it's been very interesting to see what their perspective is. And there is a lot of commonality with our philosophy, which is trying to morph education into something that is much more applied and much more ready for the real world. I'm quite interested in the name itself, like half stack. Why are you calling it half stack? I mean, is that the opposite of full stack, right? Why data science is half stack?
So probably you can explain a little. Bit no. That's a good question. It it's a response to the full stack idea. I mean, one thing we never liked was this idea that a single data scientist has to be this Unicorn who does everything in a company. I mean that's just that's not the reality that would never function in a business, especially a solid decades old enterprise. You can't just drop a single data science Unicorn in there and hope that they'll make the company lots of money with a
machine learning model. Immediately I. Just remember there was 1 presentation. We gave somewhere and Sean we talked about things like the difference between academic data science and business data science. And just somewhere on those slides he coined the phrase half stack data science. You just put it on the slide and it's sort of it just sort of
stuck. And the idea is that because data science is so different in the real world as opposed to in education, then you need people with a sort of this hybrid skill set and more generalist skill set. And I we don't really think that having someone be full stack is realistic, at least you know, being an expert in everything from data cleaning to statistics to software development to
business strategy development. I think these days is pretty hard to be full stack and even the stack gets deeper and deeper right? So I think there's so many technologies that these days people need to learn. So I think your hands probably half stack, 1/4 stack. Probably makes. Yeah, exactly. Fractional stack. So let's go into the topic of today's conversation which is about the data analysis.
So I think in the beginning you mentioned you realized there is a big gap between what normally data analysts or data scientists learn throughout their education or maybe boot camp courses, whatever that is compared with the real life problems, right. So what are the typical gaps that you see challenges for people from Academy or maybe learning from their study and thrown into the deep end into real world problems. So what are typical gaps or challenges that we have to think
about? Yeah, so I've taught accelerators and boot camps and I was faced with this problem as well of trying to teach the right skills, but also within the framework. Of what I was. Expected to teach. And so you know there's a list of technical topics that you absolutely have to teach in order to make someone a data analyst, right? You need to be able to read data, combine data, clean it, identify missing values, outliers, all this kind of
technical stuff. You know, we usually teach like Sequel, some kind of business intelligence tool like Power BI or Tableau, maybe Python, ideally have some kind of programming language in there, or maybe R See, these are sort of technical skills that you just have to teach in the foundational training, because otherwise you can't do the job right. Excel counts as well, so all all the different things you can do in Excel. And then sometimes, depending on the course, you'd also teach
machine learning. You know how to build a machine learning model, how to do some of the the sort of the practitioner, things like cross validation, other things like that. But technical skills are not the whole job, right? And I'm sure multiple guests on your podcast probably said the same thing, that software engineers don't spend most of their time writing code and data analysts don't necessarily spend most of their time analysing
data. There's all sorts of other things to do, like identifying problems to solve in the first place. That's not something we teach in foundational training. How do you have a conversation with another human and read between the lines of what problems they're actually trying to solve? And you don't even necessarily realize that that's a skill
you're going to need. And my problem with that kind of thing is that we often hand wave it away and say, Oh well, of course people would just learn that on the job, but how, right? There's no one to actually teach them in any sort of formal. Way it it's just sort of, oh, you just pick up these skills as you go. So anything about navigating like priorities in a company where five different stakeholders ask you for seven different projects, How do you
know what to work on? How do you generate a value statement for any analytical work? So how do you think about the actual quantifiable value that this project is going to have, what kind of impact it's going to have? And then some of the other things is, so you've got building all these machine learning models, but then how are these models going to be
used by the company? And so sort of planning after the first model component as well, that's something that you just sort of have to learn on the job rather than being prepared for it. And then finally, the other thing that I try and teach as much as possible is like the most realistic data sets that you can work on rather than the toy examples that we teach. Now almost everyone who's taken some kind of data science course has predicted the survivors of
the Titanic, right? That's one of the classic data science machine learning examples. And I I think the real world applicability of that problem is not that high. So the fact that the data sets in education are not that realistic is one of the problems. And the other one is often they're they're actually quite clean. You know, there's a meme in data science which is 80% of data sciences spent cleaning data and the other 20% is spent complaining about the fact that we have to clean data.
But it's true that often, you know, we're the ones who have to either collect the data or find where it is in the 1st place, combine it, document it for the first time. And these are all things that take time. And these are all things that we don't think about much in the classroom. And you know, one reason for that is we just don't have the time we have to teach these
technical skills. But I think there is room to make education better at the foundational level by incorporating more of these real world elements. Yeah, I didn't study data science, data analytics, but I did some kind of data projects. People write data reporting, real time analytics and things like that. I think the level of complexity and ambiguity I guess increases as the amount of data sources, right. And also depending on the cleanliness of the data, right.
So I think when you study in maybe, I don't know, Bootcamp or a course, right, typically you're given a a data set which is kind of like clean enough and you know, well defined and things like that. But I think as soon as you hit the real industry, right, so you realize that it's actually not as simple as that. Sometimes also like identifying the problems to solve, right, the question becomes much more
abstract probably, right? There's no, like, real formula and maybe there's no even like 100% solution that you can actually come up with. And hence the first thing that you mentioned is actually identifying the problem. So how can people who learn more about technical stuff, right? Because typically it's quite straightforward in the boot camp, like, OK, here's the data set, here's what I want you to find. And it's kind of like
straightforward. But actually in the real business world, sometimes identifying the problem is a challenge. Not to mention as well that you don't understand the domain of the business. So maybe from your experience some tips that you can teach us here. Yeah, I mean I was lucky in the the company I worked for. So we worked in the used car industry and like on I think my first day was at an actual used car auction that the company was holding.
So I actually got to see the company in operation. There's no mention of data on that day. It was all about like this is how the business operates. And I think that's a really good model. I think all data scientists and all analysts should spend more time in the business outside of the data sets, just in the actual business to see how it works.
They should be shadowing their colleagues who are in charge of either entering the data or just doing business operations, the sales people, the customer engagement people, everyone who's contributing some way to the company. I think data science should understand all those functions because then you have the context and then you understand the columns that you're seeing in the data.
Again, in education, we normally say, OK, here's the data set and then here's something called the data dictionary where each column is labeled and then we we tell you what each of the columns means, which which is great, right? For for educational purposes, great. That thing doesn't exist in the real world. Usually we have to write our own data dictionary, but even then that's a very narrow view of looking at it just as a table of
numbers. What we should look at it as is the wider context in the business, right. So where does this data come from? Who enters it, when do these records get generated? So just understanding that data generating process is really important. And yeah, you said about understanding the domain, it's really important for a data scientist to understand the domain they're working in.
And that's the kind of thing you can learn that on the job for sure, like you don't need to spend a. Year. As a used car salesman before you go into a data science job in the used car industry. But once you're in there, you know having that additional curiosity and context about the domain is definitely important and not something that you can delegate to other colleagues. Yeah, sometimes. I think also like data scientists or data analysts typically, right?
They love playing with data or their tools, their sequel or BI tools, whatever. Don't forget that you should also collaborate, right? You're good with crunching the data, cleaning the data. But if you collaborate with the domain expert, maybe sitting side by side, show the data and ask questions why this matters or what kind of data that you
are dealing with, right? So maybe if you collaborate more, you'll get to learn much better, because otherwise you'll probably won't be stuck into identifying the problem and also giving a solutions. Which brings me to the next
topic of discussion. Right in your book you mentioned this result driven approach being pragmatic when you come up with data analysis because sometimes I can see we crunch data, we use sophisticated tools and techniques, right, but doesn't necessarily bring results. You know after a few weeks of time you come up with the result, maybe the business or the stakeholders don't really get it right or don't feel satisfied with the answers.
So tell us more a little bit about your result driven approach so that we can be more pragmatic in our data analysis. Yeah, I'm glad you mentioned collaboration because that's partly what it comes down to is as data people, we need to remember that our primary goal in any company is to provide value of some sort, whether that's clearly monetary or saving time through automation. Whatever it is, that is our
primary goal. And something I say to students is if you can solve a stakeholders problem, answer the question with a single bar chart, then fine. It's the problem solving that matters. It's not the level of sophistication in your tools. And that's what I'm trying to get across in the book, is that the key thing is to have an end goal to start with. Like it's one of the first things you need to do is define an end goal that you want to reach.
And it might be like the simplest version of the problem. It might be the smallest possible answer. I call it the minimum viable answer in the book, which is you know, what is the absolute minimum amount of work that you can do to get something. A result that will take you to
the next step. And then you know, it's usually then a conversation, as you said, a collaboration with stakeholders to say, look, this is what I did based on our conversation, what do you think, what direction should we take this in? And just having that at the forefront of your mind throughout the analysis I think is really helpful because as data people, we can go down the rabbit holes. If you're working on the data set, usually you get more questions generated than
answers. Every time you look at a new column. You're like, oh, there's missing values here, oh, there's a relationship between these columns and so you can keep exploring it forever and as you say, never get to a result. So if you already know upfront what your results should be, the whole exploration has a goal that you're directing towards, which you know makes it quicker to get to an answer, and it will also make the answer more useful
and in textual. So very interesting that you mentioned we should start with an end goal in mind, right. So I think typically maybe from my experience I I didn't see many data analysis work that way, you know giving an end, end goal 1st and typically it's like stages, right. So you start with the first crunching of the data, the milestone and then you give the preliminary result, right. But I think, understanding the end goal, how should probably
the result look like? Confirming with stakeholder. That's what you advise in your book actually, right? So you do the first iteration, do as minimum as possible, which is the minimum viable answer that you have, and ask for feedback and then shape from there, right? Rather than going the rabbit hole. I think many people, especially for me when I work with data, it's always fun to crunch data, you know, firing different sequel statements and wait for the results and storing it to
somewhere, right? It's always fun and maybe generating reports or charts which are fancy, but I think sometimes it doesn't solve the problem. So I think it's very very important. Have a goal in mind, the end goal in mind. Clarify that with stakeholders and iterate right. The iteration here. How? How would you suggest people to do right? How short should the iteration be? Or what is too long for you so that you have to be wary about? So maybe a little bit of tips on the iteration.
Yeah, that's a great question and and something that we often wrangle with on our podcast. We had a whole episode dedicated to estimating time because in software world what I was used to is that we got pretty good at estimating how long a task would take, right? It's like, oh, we need to add 2 buttons to a web page and you're like, OK, I need to write some functions in the background, maybe I need to like create a new database column, I need to write this code.
It's probably going to take me a day, and we were pretty close most of the time for these little chunks of work. In the world of data analysis, there's so much uncertainty, partly because the problem is ill defined, partly because we don't know the data very well, partly because we could find anything of interest and go down all these rabbit holes that it becomes very difficult to estimate how long something will take. But you can't just say that to a
stakeholder, right? You can't just say, I have absolutely no idea when I'm going to get back to you on this. It's not, unfortunately, not viable in the real world. So what we usually said was, again, we didn't call it a minimum viable answer at the time. We just said we'll do some work to get towards this particular answer, which we've all agreed on. Looks like a plausible first step. So in a week's time we'll report back or in a few days we'll report back.
And then rather than saying when the work will be finished, we would just check in at intervals. That's one way to do it is to just say we're going to tackle this problem, we're going to work on it for a while. This is the end goal we have in mind and we'll check in in a few days and the check in might be here it is. Here's your minimum viable answer or the check in is this question that you had actually is very difficult to answer with
the data we have. Here's the other kind of data that we need to collect before we can give you an answer. Again, it's about collaborating and keeping your stakeholders in the loop so you have shorter iteration cycles basically are what you're after. So I think in the typical software development project or maybe product development, right, many teams actually don't start with a good data design so
to speak, right. So they come from the operational point of view, you know, they just do transactions, store the data and that's it, right. So there will be tables, mostly relational tables probably, and then these tables will be given to the data analyst to derive some insights, right. So I think in your book, in all the problems that you have in each chapter, right, you always come up with the data dictionary.
I know probably it's a bit luxury in the real world to actually see a good data dictionary, but tell us really the importance of this data dictionary and is it just defining, you know, table column types and column description or is there something else beyond just that? Yeah, in the real world it is rare to have a document like that. I mean, you might have pieces of it scattered around but not collected together in something as coherent as a data dictionary.
I mean, the purpose of the data dictionary is at a surface level, to record all the different columns in the data and what they mean, what data type they should be and then what they represent. Sometimes it's because the column names are abbreviated, so it just tells you sort of what the column actually means or what what the abbreviation stands for. But deeper than that, what you want a data dictionary to tell you as well is the process that
generates those columns. So for example, just to give you an example, we had sale data from the used car industry. Every time there was a sale at an auction that was recorded and we had a couple of different columns. One of them was called the sold date and the other one was called the date sold, which just sounds like we have just this redundancy of two columns that measure the same thing, but it turned out they don't measure the same thing.
What it turned out was one of them was the date that the sale happened, so when the auction happened, but there was another date that could be in the future because sometimes there's a dispute around a used car. You know, like they buy it and they look at it and there's a scratch and they didn't see it before. And so they dispute with the vendor. Maybe they'll take the negotiation offline and agree on a different price in a couple of days and then that date get stamped separately.
And so if you have a very high level surface level data dictionary that just says this is the date sold. And then for the second one it also says something like this is the sold date, that's not useful. It needs to give that deeper context of why we have these columns in the 1st place and what are the possibilities of the different ways that the values could be filled in.
So ideally the data dictionary also talks about the data generating process and how the sort of the business operations translated to this particular data set. And I think in particular it's becoming more important if let's say you have a multi stages kind of process that derives the data, right? Hence probably these days people refer to it as data lineage, right?
Where you start from a typical business process that generates the data, but then it goes through different transformations, maybe different systems, different processes until it gets to the final sync or data, the last place where the data gets stored, right. So I think the data dictionary will be much more important because you don't just see you know the column names and the values right? Which is sometimes misleading just like what you mentioned, right? I think date.
So So date is not just a common thing, right? But it's also like probably you can see it all over different data sources because different people maybe creating the column, different teams creating a column and also maybe different department using the same term, but actually means that different things, right? Hence, probably the Domain Driven design kind of a practice makes more sense and I think not just data dictionary. In your book you also mentioned
this thing called data modeling. So you mentioned this is probably the most important step as well. Before you start data analysis, tell us a bit more about data modeling. What is this step and what should people do in the data modeling exercise? So one of the ways I think about this is that there's a good definition of data science. It's been around for a while, which is the process of turning data into information and information into insights. And it's this first step of
turning data into information. And again the terminology is very sort of blurred. But when I think of data turning into information, it means the data is whatever you have lying around. As you said earlier, you alluded to operational transactions that happen have to be stored in a database. They weren't collected with analytics in mind necessarily. They're just database records that power some kind of customer facing application. And as analysts we want to come in and analyse that data.
But in it's raw form, it's almost never usable. We need to do something to it. And when we do things like what we might call data cleaning as part of an analysis, what we would like to do is to do that data cleaning once and have a cleaned version of that data stored somewhere. Part of data modelling is doing that data cleaning once, so that the logic of the cleaning is encoded already in the data that
we use. I mean, lots of companies will have this problem where for example you have a bunch of Tableau dashboards and in every dashboard there's a formula that calculates some relevant metric but that. Calculation is duplicated across every dashboard, so if you ever make a change to it. You need to remember which dashboard it's also in and it becomes the sort of mess that you you don't necessarily have a handle on.
If you have a clean data model where that metric is already pre calculated and the dashboards just read from that clean data model, then you've got that problem in one place. So if you need to change the metric, all the dashboards will update and that's the very sort of simplistic way to look at it. And the other reason to do data modelling is to sort of capture. Business entities in the right way.
So one of the problems we had to work with was what is a customer in our business and that sounds like a very simple question like obviously business knows what their customer is, but we had different business areas that worked with individuals. So individual used car. Dealers. So they were people. But then we also had customers who were entities. And they might be again, they might be a single dealership, or they might be like a parent.
Group. And so of those different entities, which one is a customer? Well, that depends on who you ask, and it depends on the purpose that you want to use the data for, And So what you need to do then is have some sort of. Customer data model where everybody agrees on the definition of a customer and then if somebody needs to know how many customers do we have, ideally all they have to do is just count star from that table.
That's the sort of dream scenario where you've done all the business logic and all the work up front to have a clean data model that can then be analyzed much more easily. And this again, is something that we don't really talk about in foundational training. We say, yes, you need to clean your data, but we don't say once you've cleaned your data, you should probably have a clean version of it somewhere and stop cleaning your data multiple times, stop repeating yourself.
And so that's why I dedicated a whole project in the book to data modelling to sort of practice this idea of taking raw data and turning it into a specific structure which is tailored, again specifically tailored to the questions you're going to ask in the business. Specifically about data cleaning, actually this is probably like what you said, the
meme, right? 80% of your effort probably is spent first understanding how dirty your data is and then like because there are many variations, right? Sometimes it could be the user input that is probably not clean. Second thing is there's no validation in the software that captures it.
And the third thing, for whatever reason, right, people put different formats, you know, like from different systems, probably they don't have a uniform format, so they just use whatever format that makes sense for them. So data cleaning probably is something that is really, really
hard. And first of all, right, if you have millions of records, for example, you probably won't understand how clean because you might look at the first, I don't know, 100 rows and you just deduce, OK, this is typically the data, but actually there are many other columns or many other
data that you don't see, right? So maybe a little bit of tips, how can we actually do this data cleaning much much efficiently so that we don't fall into the gotcha where actually you clean maybe 50% of the data, but the other 50% is something else, you know, like a different rubbish altogether. So maybe from your practical world example. Yeah, I think what's funny is we used to have this term, big data, right, to describe data that cannot be processed on your
laptop. And you don't see that term around very much. And that's because processing power, even on individual laptops, has grown so much. And even getting access to a remote cluster that has a lot more resources in your laptop is is pretty easy these days. So I don't think we often have that problem where you can only clean half of your data and the other half you can only revisit when you run some code later or something.
I mean, some companies obviously have data that's so huge that they need special methods, but I think most cases is not the case anymore. But when it comes to data cleaning, I mean one thing I would tell students is don't try to clean the whole thing at once before you do anything with it because you're going to find some issues down the line
anyway. So again, it's just this pragmatism of figure out what part of the data you need right now and have a look at it. And there are some checks you can do, right? There are some surface level checks you can check, like what are the unique values in this column, are there missing values, are there outliers? You can do those things, but some of the more complex problems or patterns in the data that will either invalidate your analysis or require you to to redo some of your work.
You won't notice them until you start working, and so again, don't be wedded to this idea that you have to make a perfectly clean data set before you start. Working. Do again the minimum that you need and just start doing the analysis with the knowledge that you're probably going to have to go back to step one again and again. So if you would ever watch me doing an analysis, it's never like I have the analysis in my head and I just have to type it out.
It's an active process where you do some stuff and you go, Oh no, this doesn't make sense at all because I found something in the column. I have to go right the way back to the top and start again. And so you might have to rewrite some of your code. You might just have to rewrite a bit at the top and keep going. And there are cases in the book where you know, I have example solutions for each project and I go down specific particular rabbit hole that I followed to
get my particular answer. And you'll see things in there where I say, oh, it turns out this is the case. There's an e-commerce data set in there and then we have some products that are miscategorized. And in the way that I wrote the example solution, it's trying to be as realistic as possible. So I haven't done the analysis and then written it up cleanly. I sort of write it up as the real process. And so halfway through, you're like, oh, these labels don't make sense.
We need to go back and fix this quality issue in the data and before we can carry on. That's the realistic way to think about it is you will find data issues throughout the process, so don't worry about getting it perfect the first time. Yeah, I think it's worth to emphasize, right, probably 100% accuracy, sometimes it's not possible, right, especially if you're dealing with really large
data, right. So maybe some kind of percentage where you would accept, OK, maybe these are the normally kind of a data, maybe state that assumption or maybe state that signal that you can see from the data. And I think there are plenty of useful tools these days that can actually give you a sense of like a distribution, for example, in a column, how different or how is the variance
of the data inside, right. It can give you a statistical distribution or it can give you some kind of patterns that you can probably deduce what kind of data is inside. So use that kind of tools. I think speaking about big data, I think these days people want to build like a data lake in a company where you put everything together into a data lake.
Maybe in your practical experience, is there some kind of different challenges that people have to deal with dealing with data lake or maybe big data in general as well? Yeah. I think it's very tempting to take absolutely any data that you have lying around and dumping it somewhere and saying, oh, we'll come back to it when we need it. And so the technology is there to allow people to do that quite easily.
And the problem with that is that there's no thought given again to the end product of like what are we going to use this data for? And dumping stuff into a data lake because at some point in the future we might need it, is not necessarily the best approach. It can create a lot of problems down the line. And if you think about the phrase data science, half of it is the word science. So that's not how science works, right? Science.
When you need to collect data, you have a hypothesis, you set up an experiment, you actually have sort of theoretical framework to build around before you even think about the data part. And I think some of that could be applied. To sort of. The business world where we don't dump stuff into a data lake for the sake of it, we think a bit more about, you know, what is the problem we're actually trying to solve. Therefore, what is the data that we need?
Therefore, where should we store what information? Thanks for the tips. I think, yeah, because of these cloud technologies and potentially storage cost is cheap, right. So they will just dump everything and maybe think about it later how we can use the data. But I think sometimes it's not why simply because yeah, the amount of data is just large, right. And how to deal with it, what kind of insights probably is just difficult if you start with
that big amount of data. And I think these kind of challenges are quite typical in a day-to-day world. But maybe from your experience, what are the typically business problems that a data analyst should know about should equip themselves with? Maybe in your book you mentioned things like categorization, dealing with time series, or maybe What are some of the favorite typical problems that
people should be aware of? Yeah, I picked the projects in the book specifically to address topics that I thought were missing from foundational data training but that actually come up a lot in the business world. You mentioned time series
forecasting. That's one of the things I talk about a lot with students is that, you know, we usually have like maybe one session on time series forecasting and we'll teach them a little bit about how to reshape time data, how to think about time data differently from tabular data and how some of the methods are different. We don't spend a lot of time talking about like econometrics or anything, which is where. There's a lot of. Time series forecasting problems.
But I think the opportunity to forecast things in the real world is actually that there's a lot of those opportunities and it's actually much bigger than we let on in basic training. So that's why I have a project dedicated to time series data. And then there's also this other idea of working with categorical data. Now that's something that we mentioned as an aside in foundational training, we'll say, yes, sometimes your data is categorical.
And here are a couple of methods that you can use to transform that data into something else. But if you work with operational data like people filling in forms and entering things in records into a system, anytime there's a drop down, you've got categorical data. And so it it's actually a lot more prevalent than we let on. We spend a lot of time talking
about correlation. We spend a lot of time looking at distributions and things for continuous data, but we don't talk about methods for categorical data enough. And one problem with that is then you're not equipped to deal with all these columns that you'll actually see in the real world. But the other problem is that people accidentally shoehorn continuous methods into categorical data. So I even break down an example
in that chapter in the book. There's a famous heart disease data set where you're trying to predict whether someone has heart disease based on various different measurements. And like the entire data set is numeric. So it looks like, oh great, we have all this continuous data, we can just throw correlation at it. We can throw all these continuous methods at it. But if you actually read the data dictionary and going back to what we said before, you actually see that most of those
values are categories. And they're like one of them is something like the slope of, I guess like a table or or treadmill or something that was during the test. And it's not a measurement, it's not an angle of the slope, it's just one of some values. So they're not on a continuous scale. So if you start applying methods that are meant for continuous data on that column, you're going to make incorrect inferences from it. And the really difficult thing about this is that you don't get
an error message if you do that. The data analysis tools are not going to tell you. Are you sure your methodology is correct? No, because you've just said I want an average of this column, but it doesn't make sense. It doesn't make sense in that context to average that column. And so the difficulty here is to remember that you need to think through your methodology harder, because the computer is not. Going to tell you otherwise. Yeah.
Hence, I think the data dictionary again that you mentioned is very, very important, right. Understand where the data gets generated, right, which business process, which system, what kind of inputs that can be possible, not just looking at the data and create your own assumption. So I think that's pretty dangerous.
And I think when you mentioned about prediction, there are a lot of problems that data analyst has to come up with, which is to actually derive predictions or maybe models to actually predict a result, right. And this is typically unknown problem where you don't actually know the accuracy of what you come up with. So how do you deal with that kind of ambiguity, first of all? And how second thing is that you can come up with a much better
prediction. So maybe something in the typical real world, do you do much more rapid iteration and test it in the production before you actually come back and derive a second derivation of what you did? So maybe some tips here as well. Yeah, that's a great question because prediction is obviously something everybody says they want. The question is, you know, what is the output of that work? That's something we found out
the hard way on a project. As you know, I built a predictive model for something that was from a technical point of view, it was accurate enough to use. And when we tried to put it into production, we found various organizational barriers to it, like the data that the predictive model requires doesn't arrive in time, so we can only make the prediction when.
It's too late. And then the clients that we would use this with didn't actually have the levers in their business to change anything based on our predictions. So it was a sort of twofold failure from an organizational point of view. And from then on, we were much more strict about, again, starting with the end of like why do you want us to make these
predictions. And I think as a data person, that's a question you should ask immediately when somebody says, I want you to build a predictive model or we should be able or we should be predicting this thing is OK. But what are you going to do with the predictions? What is going to change in the business? How, how are you going to respond to these predictions?
And so it's nice to have that conversation up front because then you know, your stakeholders forced to think about, OK, if we had this predictive model, what would we actually do with it. So again, it's not a technical challenge because I think the technical challenge of prediction is pretty well catered for. There's lots of libraries to do machine learning. There's lots of tips and tricks out there, but the organizational side of it is really where these projects are won or lost.
So my biggest advice would be again to have that human conversation of what are you actually going to do with your predictions first and foremost? Yeah, you mentioned something very interesting, right. So organizational challenge. So not necessarily all the time is a technical problem or data problem, but actually organizational challenge. And I think what you mentioned also very very insightful in my
opinion, right? Don't just build any predictive model as if like you just want to learn different algorithms and tools right, And use whatever fancy techniques. But actually thinking about how is the model going to be used in the real life scenario, what kind of value can be derived from there? Is it even possible to be used by the business? And speaking about predictive model machine learning, I mean the topic of AI. These days there are so many discussions about using AI to do
some kind of mundane analysis. Can AI be used also for data analysis? What have you seen in the industry typically how AI is going to change the landscape of data analysis? That's a very interesting question. I played around with the data analysis capabilities of these various tools. Some of them are more sophisticated at this point in time. Anyone listening in a few months time is going to change anyway, so there's no point naming tools
specifically. But, you know, some tools are more advanced in data analysis than others. And on the one hand, it's great to democratize the ability to say, Here's a somewhat messy spreadsheet, give me some information about it, give me some insights, give me the biggest drivers to success or to a sale or something to, you know, what drives property prices based on the spreadsheet of retail transactions, that kind of thing.
On the one hand, that's great because you don't have to put people through technical training to get there. But I think what it does create is the necessity that everybody understands how data analysis is done from a more sort of theoretical point of view, to understand what is possible, what are the limitations, what are the biases to look out for? What are the biases, societal biases that will be baked into
the data. And this is true for any output these AI tools generate, but also for any analysis that comes out, and also any analysis you do, regardless of AI or not, as you know, there's going to be these biases in there. One very, very basic example that I showed on a course was uploading survey results, right. So imagine you've done some kind of online survey and you've got a CSV or a spreadsheet of some sort of the responses that you can download from this tool.
And so yes, you can upload it to one of these AI tools and say, you know, give me some information. And one of the things I demoed was telling the AI to give me the average response time. So what is the average time that people took to fill in this survey? And So what the AI tool did is it identified that there is a start time and end time column. It understood that those are
supposed to be dates. It understood that they should be different so you can tell what the difference is and then that difference should be averaged. And so the answer we got from this, it was a realistic, it was a real survey data set. And the answer, it was like, oh, the average response time was something like 49 minutes from
this survey data. And you know, I said to the participants, you should never believe what comes out from the data analysis because you should think about, does that answer make sense in context? We were all very sceptical. It shouldn't be 49 minutes. It was a very short survey. And you go into the data and sure enough, there's one outlier, somebody who left the computer on all day. And so their particular response time was 8 hours, Everybody
else's was like 5 to 10 minutes. But when you said to the computer, you know, I want the average, it just took the mean of that column. And so that was heavily skewed by that one outlier. And obviously as an analyst you should have maybe taken the median something that's more robust to outliers and got a more realistic. Value. On the one hand, it's good that you there is transparency in these tools. It would actually gives you the code it ran to get to that
analytical answer. So you can double check and you can check it's homework. So in that sense it's less of a black box. But if you don't have the required data, literacy or statistical training, even that little bit of statistical training to understand what does an outlier mean? What is the difference between a mean and a median? You wouldn't necessarily see what was wrong, and you might be trapped into just believing the
answer that comes out. So just like with any response from an AI tool, and people just need the healthy skepticism of just reviewing the answer and checking whether it makes sense. Very interesting example that you gave, right? So I typically use AI these days to generate code, you know, like a coding assistant. I think it's much more well defined problem, right? You can actually test it straight away. Given an input you can see the
output. But dealing with data, I think it's a different kind of a problem because maybe you don't even know the answer, right? So if you just believe what AI is giving you, I think that's a danger. That's the first thing. Second thing, I think all these LLM tools is not definitive. So maybe you ask the first time it gives you this answer, maybe second time is a different thing. How can you actually work with that kind of request response model, which is probably
different every time you ask? And the third thing is the assumption that is baked in into how AI actually processes the data, right? Which is why the critical thinking aspect is very, very important. Do you think that data analysts should feel that their job is safer because of this? Or how should they equip themselves with the AI so that they can have AAI assistant that can power their data analysis process much better? I like the way you phrased it, which is is the job safer?
Most most of the time people will ask whether the jobs are going to be replaced by AII think it's just the same as with. Programming is for very basic tasks. I can see automation happening and I can see some of these tools taking some of the work away, potentially generating boilerplate code generating like I want to create a chart to do this. I'm not familiar with this particular visualization library, getting up to speed with that and and getting the right chart out of the other end.
You can definitely see AI accelerating that process, but just like with writing code. If. You don't understand fundamentally how that task is done. You can't check the outputs and you can't debug any problems. And the problems with data analysis, as we said earlier, are even trickier than with programming because you don't get an error message that you'll get some kind of answer. You can average a numeric column and still get an answer, whether or not that makes sense from a
methodological point of view. So I think the future might be that we have AI built in to help us accelerate those little bits of code and little bits of manual tasks that we might want to automate away. But I don't see an analyst's job changing fundamentally because we're still supposed to be trusted business advisors, we're still supposed to be generating value for the business, and AI is just going to be another tool in our. Toolbox to do that.
Yeah, I think in the, I don't know like in the non tech world some people predict that they can replace all people by AI maybe like a question and answer model right, where you can just give a data and then you start questioning them and they just give you an answer. But I think this is quite dangerous if you actually don't really understand the analysis and like for example it's a very simple example, you know survey
average response time, right. While you have outlier, the result can be really really different. So I think here the critical thinking is very very important, right? Don't just assume that everything AI generates is actually valid. So maybe from your experience, I don't know how much you have applied AI. So any kind of problems, like your favorite problems that can be solved by AI more effectively, Maybe you can share some of your power AI user
experience I guess. Yeah, I do have some AI examples in the book as well. I try to identify places where AI will actually accelerate the process. One of the examples there's a chapter where the the data set is actually a bunch of PDF files and so the task is to extract data from these PDFs and then do the analysis. And that's not something that we usually train people on. It's quite a niche thing. Although PDFs are everywhere, it is quite a niche thing to have to analyse data from PDFs.
Coming into that problem with an AI assistant means that you can accelerate the process of finding, for example, the right Python library. That's one of the examples I have in the book is I want to extract data from PDFs. What are my options in Python? Like I could go away and Google it when it'd take me a lot more time, so I I do like it using it for that if it's a domain I'm not familiar with.
If there's a problem I'm trying to solve where I think there must be a Python library for this so somebody else must have solved this problem in a way that I can use it. AI acts as like a super powered search because you it's not just a search query, you can actually give it some context about what you're trying to do. That's definitely one aspect I see. And then just, you know, accelerating smaller tasks. I mentioned creating charts.
If you want to figure out how to do a specific kind of data visualization, and again, you're unfamiliar with the library, you
need a bit of help. You know, I can give you the starter code for it. Although one problem, and this is I think true for Copilot and other kinds of tools that generate code, is that it's only trained on what it's found on the Internet and there's a lot of sort of not even incorrect code, but maybe inefficient code or maybe not quite the right way to do it. Again, a very specific example is there's a Python plotting library called matplotlib, and
there's sort of two different ways to use it. One of them is the old MATLAB style. If anybody listening has ever used Matlab, there's a specific way to create plots in MATLAB, which is how matplotlib was originally written, and that's how you create plots in it. But now there is a much more modern object oriented interface for building with map plot Lib where you create your chart. Object and you assign the properties and stuff like more
how you. Would sort of write software in general, but unfortunately a lot of the code on the Internet uses the old style, and so the AI might perpetuate the use of the outdated, less modern style of code. So even for tasks like using a library, you've got to be careful about what the training data is out there. That's practical tips for people, maybe some creative idea how you apply AI in your day-to-day job, right? So I think extracting text from PDF also like coming up with
different charts. I could also imagine like for example downloading data from a typical API, right? Because if you're not familiar with the API, maybe AI can help accelerate that. Or maybe just like transforming data into like a different format. That is probably also another small task that you can use to solve the problem using AI. So David it's been quite a great insight you know conversation about data analysis.
As we wrap up the conversation, I have one last question that I would like to ask you which is something that I call tree technical leadership wisdom. So if you can think of it just like an advice that you want to give to the listeners here, maybe if you can share your version of tree technical leadership wisdom. Yeah, sure. So one of the things that I think differentiates like good analysts from the best analysts is curiosity. Just wanting to find out the answer to something.
I don't know if that's good advice because I don't know if you can teach that to someone. I don't know if you can learn to be more curious. But even if just mechanically you try to find the answer to things and persevere beyond the first answer or beyond the obvious answer, that is really such an important skill in life. But it's particularly in data analysis, like really wanting to dig in to find the answer and not resting until you're satisfied with the answer is a
particularly good skill. And just from that, I think practicing your skills, just whatever your tech skills are, just constantly practicing them is so important. I mean, I'm just writing, you know, this book is a project based book full of practice opportunities for people. Because I think that's really once you've got the foundations, the best way to learn is to apply your knowledge and practice. And so just keep doing that in
the data realm. That just might mean solving problems for yourself, even if it's optimizing your fantasy. Football team or whatever. It doesn't have to be a money making business opportunity every time. Just something that solves a problem with data is great. So practicing your skills to stay relevant and to keep them fresh and to learn new things is vital. And just on the sort of more organizational side of things, I think it's very important to have a good reason.
To do data. Work in the 1st place. I think there is a maybe mistaken belief in business that doing data analysis is inherently useful. It's just inherently a good thing that we should be doing. But if you don't have a purpose, if you don't have an end goal, if you don't have a good reason to do it, it's not going to be successful. And that's true for analysts as much as entire organizational strategies just have a good reason to do data work in the first place.
Very interesting. Last wisdom there. So I think I can actually relate with that kind of advice that you just gave, right? Because many people think, OK, we have the data, we have the data. So let's just come up with the insights. What insights are? Maybe they don't know what kind of insights they want to derive from it. So I think, yeah, know the reason why you want to tackle data problem. I think it's really, really
important. So for people who love this conversation, they want to learn from you further. Or maybe they just want to find more about yourself and your book, maybe. Is there a place where they can find you online? Yeah, I think probably following me on LinkedIn is probably the best place to look. So my name is pretty uncommon, so you can put it in the search
bar and find it pretty easily. Yeah, I don't post much on other social media outlets anymore, so I think LinkedIn is probably the way to see what I'm doing more day-to-day. I also have a website where you can check out the book and there's a link to the podcast and other things that I've written. And yeah, the book is available on the Manning website, and when it comes out, it'll be available on Amazon. Thank you.
I wish you good luck in the process of publishing that, and I hope people today who listen have much more equipped into solving any data analysis problem, just like the title of your book. Thanks Henry the.