Despite innovations in data architecture, infrastructure, and analytics, most organizations today still struggle to realize the promised value of data. Learn how the data mesh principle of data as a product can help, as part of a data mesh initiative or as a stand-alone strategy. Published at: https://www.eckerson.com/articles/data-products-part-of-a-data-mesh-initiative-or-a-stand-alone-strategy
Jun 14, 2023•10 min
Data mesh is a new paradigm for fulfilling the promised value of data. It decentralizes both data ownership and the data itself, shifting them toward the functional domains that create and use data to operate. But data mesh is not for everyone. Learn how to assess if you’re ready for data mesh. Published at: https://www.eckerson.com/articles/data-mesh-evaluating-your-organization-s-readiness-for-a-decentralized-data-future
Jun 14, 2023•10 min
There’s so much hype surrounding data products that you have to wonder if it’s just another buzzword. But there’s more to data products than buzz. In this article, you’ll learn how the concept is a meaningful step forward in the art and science of data management. Published at: https://www.eckerson.com/articles/best-practices-for-developing-and-scaling-data-products
May 24, 2023•9 min
A zone-based data refinery creates an agile, adaptable data environment that supports new and unanticipated business requirements quickly. It turns a monolithic data warehouse into a flexible data environment that gracefully adapts to new and unanticipated business requirements while maximizing reuse and standards. Published at: https://www.eckerson.com/articles/how-zone-based-data-processing-turns-your-monolithic-data-warehouse-into-a-flexible-modern-data-architecture
May 24, 2023•6 min
Many data engineers already use large language models to assist data ingestion, transformation, DataOps, and orchestration. This blog commences a series that explores the emergence of ChatGPT, Bard, and LLM tools from data pipeline vendors, and their implications for the discipline of data engineering. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-examining-the-role-of-chatgpt-and-large-language-models-in-data-engineering
May 24, 2023•8 min
At IAPP Summit, privacy and data governance leaders expressed the importance of a collaborative operating model. Published at: https://www.eckerson.com/articles/the-convergence-of-data-governance-and-privacy-takeaways-from-the-global-privacy-summit
May 24, 2023•7 min
Embeddings are a learned way of representing data in space. Vector databases make it easier to work with embeddings generated from deep learning models. They will become an essential tool in the AI stack because they reduce the time to structure data and train models. Published at: https://www.eckerson.com/articles/the-why-what-who-and-where-of-vector-databases
May 24, 2023•10 min
A robust data workflow testing strategy helps ensure the accuracy and reliability of data processed within a pipeline. Use this checklist to meet your organization’s data quality requirements according to the dimensions of accuracy, completeness, conformity, consistency, integrity, precision, timeliness, and uniqueness. Published at: https://www.eckerson.com/articles/developing-a-robust-data-quality-strategy-for-your-data-pipeline-workflows
May 24, 2023•7 min
An operational data hub (ODH) is a pattern in data architecture that provides a central location and a standard protocol for operational systems to communicate about and share data among themselves. Operational systems post messages about data events (add, change, delete) and subscribe to messages of interest posted by other applications. The hub works to share data among applications without the clutter and chaos of point-to-point data feeds. Published at: https://www.eckerson.com/articles/oper...
Apr 12, 2023•7 min
Data mesh is a hot topic in the data world, generating conversations about the benefits and drawbacks of its decentralized approach. Concerns about an explosion of data silos and inconsistent data quality are justified. But to those who feel a bit like Chicken Little, maybe the sky is not falling. Published at: https://www.eckerson.com/articles/data-mesh-the-sky-is-not-falling
Apr 12, 2023•8 min
The data mesh paradigm is in a nascent stage with data personas and organizations craving clarity and quick answers. Best practices are yet to be crystallized. Mesh done incorrectly runs the risk of degenerating into silos. Published at: https://www.eckerson.com/articles/caution-data-leaders-plan-carefully-before-rushing-to-data-mesh
Apr 12, 2023•10 min
I got energized walking the show floor at the Gartner Data & Analytics event last month and learned a few things about the future of our industry. Published at: https://www.eckerson.com/articles/quick-recap-of-gartner-conference-2023
Apr 12, 2023•8 min
The modern data stack is a loose collection of technologies, often cloud-based, that collaboratively process and store data to support modern analytics. It must be automated, low code/no code, AI-assisted, graph-enabled, multimodal, streaming, distributed, meshy, converged, polyglot, open, and governed. Published at: https://www.eckerson.com/articles/twelve-must-have-characteristics-of-a-modern-data-stack
Apr 06, 2023•10 min
AutoML and the emerging approach of declarative ML help simplify the process of creating and refining ML models. Published at: https://www.eckerson.com/articles/automl-and-declarative-machine-learning-comparing-use-cases
Apr 06, 2023•10 min
One version of the truth is the holy grail of data and analytics. However, the promise of one version of the truth still evades us because even with consistent data, the truth is, as the film My Cousin Vinny demonstrates, a matter of perspective and context. Published at: https://www.eckerson.com/articles/one-version-of-the-truth-according-to-my-cousin-vinny
Mar 20, 2023•6 min
Data leaders know the importance of change management, but few understand the dynamics involved in driving adoption. A new book by Damon Centola shows how social networks spread and inhibit change. Published at: https://www.eckerson.com/articles/how-change-happens-driving-technology-adoption
Mar 13, 2023•8 min
Over the past 20 years or more, data architecture practices have focused almost exclusively on managing data for analytics. Operational data is much more than source data for analytics. We must give attention to operational data architecture or pay the price in data disparity, data friction, and technical debt. Published at: https://www.eckerson.com/articles/operational-data-architecture
Mar 08, 2023•11 min
Designed and implemented well, automated workflows can make the modern business just a little less chaotic and complex. This blog explores the opportunity for automated workflows to help cross-functional teams collaborate and standardize organizational master data. Published at: https://www.eckerson.com/articles/master-data-management-and-operational-workflows-two-modern-use-cases
Mar 08, 2023•6 min
Data fabric is one of those buzzwords that’s used so much and in so many ways that it often elicits an eyeroll—undeservedly so. The phrase is shorthand for a complex and important set of issues that we’re all working to manage. In this article we’ll review what data fabric is and why it’s important. Published at: https://www.eckerson.com/articles/data-fabric-s-use-of-abstraction-and-metadata
Feb 14, 2023•9 min
The data pipeline market comprises four segments: data ingestion, data transformation, DataOps, and orchestration. This blog defines three principles for successful pipelines: (1) watch the innovative startups; (2) use suites where you can; and (3) use point tools where you must. Published at: https://www.eckerson.com/articles/modern-data-pipelines-three-principles-for-success
Feb 10, 2023•8 min
The data mesh framework doesn’t specify a key component that completes the last mile of the architecture: a data provisioning environment. New technology that underpins modern data marketplaces complement data mesh, providing a frictionless way for data providers and data consumers to exchange data. Published at: https://www.eckerson.com/articles/data-mesh-s-missing-ingredient-a-data-marketplace
Feb 09, 2023•8 min
Traditional techniques for managing data quality break at scale. Machine learning algorithms can automate aspects of the data quality workload, ensuring that the data the business users consume is reliable. This article profiles three tools and approaches that use ML to automate data quality. Published at: https://www.eckerson.com/articles/three-data-quality-automation-tools-you-should-consider
Jan 23, 2023•10 min
We must treat metadata like a fully-vested member of the enterprise data landscape. A unifying taxonomy is a good place to start making metadata a focus of data management rather than just a tool. This article explores how to start wrangling diverse and distributed metadata. Published at: https://www.eckerson.com/articles/wrangling-metadata-making-it-the-object-of-data-management
Jan 18, 2023•11 min
We enter 2023 in a haze of uncertainty. Enterprises must rationalize analytics projects, shift to lower-risk use cases, and control cloud costs. They also must measure the ROI of analytics projects and use data governance to reduce business risk. Published at: https://www.eckerson.com/articles/analyzing-a-downturn-five-principles-for-data-analytics-in-2023
Jan 18, 2023•6 min
This blog defines governed data integration and describes how it enabled two manufacturers to synchronize data flows from the factory floor to the customer. Published at: https://www.eckerson.com/articles/governed-data-integration-for-manufacturers
Jan 05, 2023•7 min
A rising number of financial services firms are adopting the discipline of governed data integration to build 360-degree customer views. Published at: https://www.eckerson.com/articles/governed-data-integration-for-financial-services
Jan 05, 2023•7 min
Synthetic data and artificial intelligence (AI) complement each other but are both subject to the risk of AI bias. Consequently, companies need to implement architectural and governance controls to reduce the bias that synthetic data can inject into AI models. Published at: https://www.eckerson.com/articles/mitigating-the-risk-of-bias-in-synthetic-data-for-ai
Jan 05, 2023•9 min
As enterprises grow more dependent on the cloud and as the economy convulses, FinOps will soon become mandatory. Published at: https://www.eckerson.com/articles/the-rise-of-finops-cost-governance-for-cloud-based-analytics
Jan 05, 2023•7 min
Business domains have a range of data & analytics capabilities that enterprise data teams must support. The key is to ensure domain activity aligns with enterprise standards and best practices to ensure data consistency and avoid silos. Published at: https://www.eckerson.com/articles/an-operating-model-for-data-analytics-part-iv-red-team-composition
Jan 05, 2023•8 min
Active metadata is not a type of metadata, it’s a way of using metadata to power systems. Active metadata is a critical feature of modern data architectures such as data fabric and data mesh. It makes things work such as data access management, data classification, and data quality management. Published at: https://www.eckerson.com/articles/active-metadata-the-critical-factor-for-mastering-modern-data-management
Nov 21, 2022•6 min
