The Better Business Analysis Institute presents the Better Business Analysis podcast with Hi everybody. It's Ben Walsh from the Better Business Analysis Institute. And today we are going to be diving deep into data or data analysis. The, I guess the era or the capability or process we're talking about is data management. So if you're a BA who likes to think about things in terms of groups of processes, this is the whole area that talks about data management.
Now as we've already noted in our Generative AI podcast and earlier on when we talked about different disciplines that have spun off from business analysis. Data is really important. So data science, data analysis in the true sense, I guess anything to do with machine learning and big data are probably words that you have heard. And if you haven't heard of those words, you may have heard
of the tools that use. That new kind of data architecture which is ChatGPT for example, or actually anything these days that you that surface from the likes of Google or Microsoft or Amazon are using these technologies. So they are not new, they've been around for a while, but we're now at a point we are even smaller businesses, government departments are moving towards this. Cloud is usually the word that's in the front.
It doesn't necessarily have to be cloud, but cloud data architecture, which means that the way we did things on premise or you know with database servers back in the day, it's starting to change for all of us. So it's important for us as Bas to really understand what that looks like. So we're going to take a bit of a holistic view, we're not going to go. Too deep. And I'm going to actually call out where I think the line is here for Bas versus those in the
data specialized roles. So you know, you know how how much information you do need to know and then if it sparks your interest to know more than you can of course go below into the technology side and you know, understand what each component does. So yes, this episode is around data management and let's get started. Okay. So I guess there are three ways in which we could look at this topic. We could look at it from the bottom up.
The one way of talking about that would be the architecture or the data model. That's one way we could look at it or we could look at it from the kind of how it. Is exposed to our users in terms of our processes and functions, in terms of what we call capabilities. Or we could talk about it holistically at the top from the data governance going down to the framework. Now as Bas, it's always best to start from the top down.
Yes, bottom up or playing chess and working out what's behind the curtain is always useful sometimes, so we can test our assumptions. But at the end of the day, especially in this topic, if you haven't started from the top down, you won't get the results that you need. This is very, very similar to projects.
When objectives or problems aren't defined, you know it becomes a mess, or you know the projects become less and less likely to be delivered if you haven't got that up front kind of managed up. Governance and framework in place before you start. So that is true of BA work in terms of the BA plan, but it's also true of the data side.
So without going into boring detail around meetings and data governance boards and all the rest of it, which is part of this, let's talk about the fundamental elements of data management. Data management does incorporate what I've just said, the data governance layer, let's let's not go into that box, it's not very, I don't know, interesting
to talk about. But there is a you need to have a governance structure and processes then in the kind of functional side or if you like the operational side, there are two main parts of data management. And you could say that they are defined in a data management framework. The two parts are the data
management lifecycle. So the lifecycle in which data projects go through or a piece of data goes through, they're one and the same and they are a structured way, a lifecycle if you like, of. Defining the steps in which your data project should go through and you know, stage gate and be thought about and also a piece of data goes through this process and we're going to refer to some really useful predefined lifecycle Data management, Lifecycle terminology and
journey and steps referencing data dot. Gov dot NZ, which is the New Zealand government data kind of experts. It is one way of defining this journey. You may have come across others. There's, I think there are a lot of data bodies of knowledge which define what this life cycle is. I'm going to refer to this one because I quite like it and it's relevant here in New Zealand. So if you don't have one and especially if you live in New Zealand, I would suggest just
adopting this life cycle. So that's that's a big one straight away and I'll define the steps and what they meant in a minute. So we've got our life cycle. So this these are the steps in which data go through and your projects that have data in them should go through then. So if you like, that's the process. Then in order to support you doing that and in order to support data management, there are capabilities.
There are different skills, human capabilities, but technical capabilities that need to be in place in order for you to have true data management. Now those capabilities would have been quite different 20 years ago or 40 years ago. So when we talk about these technical capabilities, these data management capabilities, they are the, I guess, the architecture, the data architecture.
That exists in 2023, So if you have been involved in, you know, you've been a DBA or you've worked with databases, this might be an evolution of what you've already known. And some of these words will not be dissimilar to the standard pattern for managing data, but below the capability. When we actually look, you know, into the engine, the technical architecture, those technologies have definitely changed.
So we've got our lifecycle and we've got our data capabilities and that's what we're gonna be focusing on today in this podcast. As I mentioned earlier, above all that is data governance. So that's the more management side of data in terms of committees and organization.
And in terms of just making sure that you've got that governance across the top and then below all of this, below the life cycle and the capabilities, you should just have some data fundamental management and that is just best practice, things like data security classifications, just very basic ways in which people. Manage data. So they're not specific to the capabilities necessarily or the
steps you go through. They go across all of those and they're just basic ways in which we deal with pieces of data, Okay. So you've got the foundational, if you like, fundamental capabilities down at the bottom. You've got data governance across the top. But what we're talking about is the middle, the actual when the rubber hits the road, which is the life cycle, which data goes through and which your projects should go through when they deal
with data and capabilities. Now capabilities can be talked about in terms of from a BA perspective and that's what we're going to touch on. And then below those capabilities, you have the architecture. So we use capabilities as a way to talk about both the human and technology capabilities that are needed. And below that, that's the connection back to the architecture. Back to the solution, the Hal. OK cool, so let's get started on this data lifecycle now, the
data management lifecycle. As I said, I am going to refer to data.gov dot NZ for those who want to reference and see what I'm talking about here. And I'll take these on on on a blog, on our website on the Better Business Analysis Institute, a website as well so you can reference it. Now when we go through this lifecycle, it's made-up of seven steps, so not too many for you to understand. Usually 5, somewhere between 5:00 and 8:00.
Things are generally a good amount of steps, which is why I like this particular journey map, if you like. The first step is plan. So this is less technical and more around, you know the processes and resources that are that you need to manage the data. It's around the project goals for this data and just really making it really clear, you know, we can use something called the data management plan if we really like.
It's where you create, you know how you're going to handle this data, what's the plan for this piece of data or pieces of data. So we're not just talking about the name. On a customer, we're talking about customer data or we're talking about moving a whole server that has a whole lot of data sets to a new technology in the cloud. So how are we going to handle that? What is the plan? So that should be, you know, you should work with your project manager on that and that should
be very clear. But it's specific around the data side of things and it might be how you're going to migrate data. You know, what is the plan before you even start doing anything. So it doesn't necessarily involve any. Fundamental data management capabilities. At this stage it could just be pen and paper or a Google doc or you know, a Word document or a PowerPoint presentation, but you're really planning out what you need to do.
So you know, you would start with writing plan and then you would list out what are you going to do with that. Now it's really important that your stakeholders, your product owners, your sponsors understand what the plan is. So it should be written in their language. A bit like a spec, but just the same as an agile BA plan that we talk about at The Better Business. Analysis Institute. This is a plan for your data,
right? Cool. So you've got your plan and now you're going to execute on your plan. So the second step after plan is collect. So this is where data is gathered or generated. And this is the information that's in scope of your plan. So you need to be able to collect this information. And it's not. We're not. You may start. Your brain will be triggered and you will start to think about
how that's going to happen. But at the moment just identify the different datasets which you've got in your plan. And we need to, you know, we need to collect information. So this could be information and other data stores within your organization. It could be accessing or consuming data from other locations outside of your organization. It could be multiple data sources. It could even be capturing new
data through say survey forms. So it's all of that and you just need to define, you know, what are your collection points, how are you going to collect this data, what is that? And literally you need to collect first. That is the first you don't. If you don't have the data, you need to go and collect the data. So it's about gathering that. And you know when we get down to the capabilities, you know there is technical terms like ingesting that data.
So how do you get that data into your particular data management platform. So at this stage you're collecting it, where it goes is important, but we're not talking about the how right now. So we've plans got a plan, we then collect now as the data comes in from multiple sources. We need to describe the data. We need to have some agreement around the metadata standards in our organization and we need to describe the pieces of data that we've got in.
So you know, you may and your plan come up with a data model. So we're saying okay, we're going to collect information around our customers. We're a store. We're going to collect the information around our store and we're going to collect the information around our inventory and we might, I don't know, sell tshirts for example. So therefore you could have grouped that data into those three areas in terms of inventory, in terms of store and
in terms of customer. And you could have mapped out what data you need to collect, only collect data you're gonna use and you would have defined that. So at this point, you may have lots of data sources you may have downloaded. I don't know, bought e-mail marketing lists with a whole lot of customer names who might be
interested in your T-shirts. You may know what the inventory system fields are, so you know what those are and you may also know all the information you need to run the shop. So you've defined that you've run that down and that's what we talk about and it's important down to go the down to the data type level here, not database level, but just know that a name might be. Just a string or a piece of text, for example. Whereas if we were collecting a phone number, that would be a number.
So it is worth, you know, describing that data at that level at least as a BA, and passing that on to. Those in the data team to maybe go a little bit further or for you to do that later on if that happens to be your role. So yeah, describe the pieces of data, what are they, What are they used for, what are the, what would they, where did they come from, what are they going to be used for? So input and then the output, not necessarily the process.
Of course. So now that we've collected our data, if you like, or at least virtually collected our data and we've named and appropriately defined the metadata around that data describing it, we then need to store it. So the data needs to be stored somewhere. And data is digital these days, mostly digital. And if it isn't, we need to get it into digital form and we need to be able to store it in a digital repository. And that's the highest term, that's a very BA term for.
Saying you know what you might think of as a database for example, or data like, there's a whole lot of terms of ways you can store information these days. But we are talking about at the highest level here, digital repository. And that of course needs to be secure, reusable. You know, we need to be able to protect the data we're collecting, especially if it's customer data. But equally if it's around our operations, around our store and our inventory, you know, it's
not. If we, you know, don't store that appropriately, we put it into, I don't know, into save all of our information into a Google Sheet that's got a public link and that information is accessed, then you know that could have negative consequences both to our reputation but also to our business and to our store, so. We really need to, you know, think about how we're going to store information and that that quickly follows the collection.
So you're collecting it. You've you've obviously got to put it somewhere. So this is and you might say, well, Ben, why are we just thinking about this now? Well, you're not, you've done this in your plan. You've already got an idea about where you might have stored it, but right now it's the action of storing the data somewhere. OK. And that could be multiple places. We'll come back to that when we talk about technical capabilities.
Once the data's stored, so you've collected, you've described it, and you've stored it somewhere or in multiple places, you then analyze the data, Okay and so that is, you know, you explore it, you interpret it, You look at the data in different ways. It might be the number of sales for T-shirts in your store, for example. It might be the number of customers who visited the store. It might be the number of transactions, it might be your
stock levels. So you're analyzing the information you're really looking for, grouping that information into useful. So when you're analyzing, the goal of it is to interpret it into useful data sets or data views if you like. So you might combine data from your customer repository. And your inventory repository and your store repository to have, you know one. So you are able to create a view where you see all customers by store, by inventory for example, where they bought their stuff.
So the next step, so once you've analyzed the data and you should have an idea about how you want to use your data at that point and then you want to use it. So that's the next step. So we've planned, we've collected, we've described, we've stored it, we're analyzing and now we're using it. The data is used for a purpose. It was collected or generated. So you should have defined why you were collecting the information and now you want to use it.
It might be a report that that's a general use case, is that you are collecting information so you have better insights on your customers and so this could be a report or you know, a business intelligence dashboard if you like. Dashboard is a very good use of data and you know the idea is that you could reuse that data also for for other purposes. You might feed it, now you've collected it and analyzed it. You might feed it into your e-mail marketing system.
So using can can be consumed is another way of expressing that. It could be to present the data inside report, but it could also be make it available for use to be consumed. By other systems. So it could be, yeah, like I said, your e-mail marketing tool. So that'd be very useful to be used by MailChimp or you know, the meta platform or whatever marketing tool you're using. And the final step is to then save.
Or destroy the data. So if you're going to keep that data for a long time, and you know this is where records management policies come in, this is where you really need to understand archival rules and disposal rules and how long you're allowed to keep that customer data for. You need to make a decision here in terms of saving it, which is could be archived because it might not necessarily need to be accessed all the time, or you need to destroy it.
You need to take actions to safeguard that data for its long term. Viability and availability. So in practical terms, that data might not be, you know, saved on your desktop and might be saved in the database. So this is we really need to think about how we get a long term save our information. Now just to recap, when you've done your plan at your first planning stage, you should have these headers written down. Click describe store.
Analyze. Use. They will destroy and you've already mapped out how you're gonna do those things, at least at a high level. You know high level business requirements level, not necessarily the detail necessarily. And as you go into each level, each step you may define the detail level requirements. And the solution you're going to use, so that is the data management lifecycle as defined by data.gov dot NZ. It is pretty good. And the other useful side of this particular plan is they do
talk about digital capabilities. Now when I talk about capabilities, capabilities is a word that is used a lot you can talk about. It means skills, and you can also mean you know human capabilities or technical capabilities. data.gov dot NZ outlines the human capabilities, the processes or abilities that your particular team or people in your organization will need in order to kind of reach a high level of maturity. And so it's if you if you if you
have that requirement. If your job is to audit where you're at, please go to that website. It's fantastic and it will it aligns those capabilities with the data management lifecycle I've just talked about. So your your job is is nearly done but because we live in an IT world and our job is to. Jump between the world of business and IT we we should be really talking about what I would say the technical capabilities that we now require in order to support that life cycle.
And like I said at the start of this podcast, these capabilities have evolved over time. They to be honest these highest level capability names may well have existed 20 years ago, but definitely the techniques or some of the ways in which we do things inside these boxes have changed. So inside these boxes there are sub capabilities if you like, functions or features that these various systems allow us to do or do well. But at the highest level, we as Bas need to think about these things.
They're not complicated and I'm going to read them out. Now this particularly are names that really Amazon AWS use to define these areas, but I think they're pretty common across whatever platform you're using, be that Google Cloud or Azure or even an on premise setup. OK, so these capabilities are technical capabilities. Just to recap, they need to exist in order to support that data management lifecycle I just talked about. The first one is data sources.
So these are things. These are nouns. They're not just collect data or store data. They're actions. This is data sources. We need to have data sources. And that could be a database, it could be SQL Server, it could be a other systems like CRM systems. It could be devices these days, It could be small devices that feed in, it could be your contact center, it could be logs, it could be anything, right? It could be your social media marketing platform. But data sources need to exist.
You need to have some way of storing, sorry of sourcing information. You need to know what those are okay, so data source is pretty easy. Then in order to use that information, we need to get that information into a platform or to a series of components. So it's useful for us and we call that function ingestion Okay. So we ingest data. If you like, you could say capture is probably the very high level, you know, holistic word, but it's not quite right.
In this case, the reason why sorry, ingestion is really important is that we're actually taking the information and putting it somewhere now that's that's important. There's an important reason for using that word because it's suggesting you're taking a copy of that information and there are other ways of accessing information.
You don't have to move it. You could report directly from a source system or you could use something called data fabric and data virtualization to kind of pull that information when you needed. It but not ingest the information. OK. So if we're going to, if it's really critical for us and we're going to add we want to tag it and we want to use it effectively and we want to process that information and we wanted to join it with others, then it is best for us to ingest
that information. So we've got data sources and we're going to ingest at least some of that information into our system. We then need to store that information. So we've got storage. Now generally there are three categories within there. You've got you ingest yours saving what you've ingested and its raw form. So whatever format and what you've got it that we then then do a process of cleaning the information and we curate the data.
So there are storage involves kind of moving information around. You know, processing that information. So there might be different levels of kind of copies of that information in various states. So we need to have storage now as I just mentioned, we need to have processing and this is where advances in data management have really taken off. So we can you know in terms of AI, machine learning and some of the tools that we've got available, the way in which we
clean data. This is part of processing and we transform data. The tools out there are much you know are really advanced. And this is where you know the old ETL processes have really been replaced by using some systems that can do that automatically. So if we take raw data and our storage, we may process that data to clean it, transform it, and then we've got a clean data set. So regardless of what format. The data came in after we've cleaned it and transformed it.
We get it into a clean, useful way, and then we go back and we compress this a bit around, aggregating information, segmenting it. We might have multiple. Different copies of the same piece of data. So it could mash that up and we can enrich the data and then we kind of have this saved back curated data. And again this is kind of a almost a modern day processing
technique. So if all our data, if you think about it, all our different sources, we had 20 different sources, we inject ingest those, they're in raw format, we clean them and transform them so they get into our data model. We then enrich them and we. Segment them. So now it's in more than just our data model. It's in a useful format. Then we can start to use that information and we use that information in a couple of ways.
One, we use analytics. So data analytics, this is where the term data analysis came from is when you're starting to use the information and you can think about that and through statistical data analysis, data science and dashboarding. The other way we could use it is we could share it with others or collaborate with other systems. So we were just doing this, use this platform for a part of our business. Then we could provide an API so it could be used by another part
of the business or another tool. And we could also collaborate with other partners or stakeholders or agencies that want to use our information. So they're two consumption use cases. One is analytics, presenting the information and one is collaboration where we can share the information so other people can consume it.
Obviously what we also need to think about is there are some more technical kind of access and security capabilities we need, which is roughly called technical data governance, but we need to have a capability for access rights.
And in terms of controls and auditing and security and then there is another bit which is really important which is called cataloging where we assign, we have a kind of a metadata schemas and potentially data crawlers that will start to tag our data automatically so that they can be used more effectively and kind of self managed within the system. So just to recap, we've got data sources, we ingest those, we've got storage for those, we've got processing.
Then consumer either through analytics or collaboration, we then have some kind of security access audit capability, which you could call technical data governance and then we've got cataloging. And if you happen to be a I guess a social marketing company or a or an advertising company, you may what we what we term activate that data. So you could use that specifically for advertising or real time marketing, but most companies don't have that piece.
So those are the technical capabilities that need to exist.
And if you have those capabilities and you've planned and you have a way of saving or destroying your data, then those middle capabilities that we talked about before, sorry, those middle process steps of collection, collecting, describing, storing, analyzing and using data will be fulfilled by those technical capabilities we just talked about, which were data sources ingestion, storage, processing, analytics, collaboration, cataloging and some kind of data.
Technical data governance in terms of security and audit and access, OK, so those are the two main parts. If you have all of those pieces, the process and the capabilities, then you have a modern day, well, I guess you have at least a framework for data management. What then makes the difference here is what is doing or what is what technology is performing those capabilities.
So I mentioned before the likes of AWS, there's Google Cloud, there's Azure, there are many others and they all have their own flavor of tools that carry out those capabilities we talked about. So for example. AWS in terms of their customer data platform, they will suck in many or any data sources and ingest those and it uses things like Amazon Kinetics and it uses Appflow or Amazon API Gateway and then stores it into buckets which called S3 buckets for
example. It then uses step functions with the process and orchestrate the information. It uses AWS Lambda. AWS Glue for workflows and then pushes that information out to analytics through Amazon Redshift. It even has its own reporting tool. And for data collaboration it uses gateways again and cataloging. It uses AWS Glue for data cataloging. So Amazon has subcomponents, technical pieces of the puzzles or services that it. And allow which allows you to
fulfill those capabilities. So Sba's, we don't necessarily care what those technical functions necessarily are, features or tools, sorry that we don't really care what they are. We just need to know that we have a way of accessing our sources, ingesting, storing, you know, processing that information and consuming consuming it. And cataloging it. So if we have those in place, then we're not. We don't necessarily need to
understand these technologies. And most of the time, there's usually a hybrid approach where some of this information is still on premise and not in the cloud. Or you're using AWS to do part of this. And then when you do your reporting, you're using Power BI and it's connecting to this. So it doesn't necessarily mean you need to adopt one whole vendor to do all these things, which might become expensive. However, all those capabilities need to exist. So what does that mean for you
as a BA? Now you understand that there needs to be this data management framework which includes the life cycle and these capabilities and means that you can actually start to write requirements, start to plan, put together a data plan and start to write requirements that use
these features. So for example you might say as a user, sorry, as a Yep as a data analyst I would like the New Zealand Census household income data to be ingested into the data warehouse and therefore you can use these terms which means that the if you've grouped those under these kind of features and functional areas. Then the architect then knows that you've got a requirement to use those areas of components, and then it can start thinking about technology that might be
best fit for that purpose. And of course, depending on how many data sets you have, how complicated they are, that could determine what kind of tools they use in those areas. You can ingest data, for example into Google Sheets or into Excel, and so you know that might be OK as long as you have. The appropriate data governance and security and access rights, which is why we don't really use Excel for that purpose. But that might be OK and that might be fine for us depending
on the use case we've got. So I hope that that's given you some insights in terms of how data management works and what you need to think about as a BA. I'm sure there's lots of other subtopics we could go into, but I hope you have enjoyed this podcast and I'll see you next time.