Why the Data Mesh fits with the Data Platform
The Data Mesh is not a piece of architecture or software. A Data Mesh is a framework for organising your business across technology and data owners.
A data mesh aims to decentralise and democracise data management from a central team to specific domains. It treats each data set a product with its own autonomy and therefore its own ownership.
It is built on 4 core principals :
Domain-Oriented Decentralized Ownership: Data is managed by the teams that produce and consume it, rather than a centralized data lake or warehouse team.
Data as a Product: Domains treat their data sets as products, ensuring they are high-quality, discoverable, and easy for other teams to use.
Self-Serve Data Platform: A central platform team provides tools and infrastructure that enable domain teams to build, manage, and consume data products autonomously.
Federated Computational Governance: A governing group with representatives from all domains sets shared standards for interoperability, security, and quality, which are enforced automatically.
The main concerns with a Data Platform is there is resistance to the idea of creating a centralised team which can not service the huge demands from all departments placed on it. The common view is that the central Data Platform team expands so much that it collapses from the weight of work and conflicting demands.
The Data Mesh framework means that the central team is small and agile, each of the teams are responsible for their own data – loading, running quality checks and corrections. The central team is responsible for the providing the teams with tools and patterns. It
- breakdown the bottlenecks with a central team,
- encourages parallel development, and
- improves data quality