Learn about building a data mesh on event streams with Apache Kafka® and Confluent.
Data mesh is a concept that ensures data access, governance, federation, and interoperability across distributed teams and systems. First introduced by Zhamak Dehghani in How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh, data mesh aims to solve scaling failures in centralized approaches in data systems. As companies become software, there is a general trend towards decentralized data ownership to enable scale.
Data mesh architecture is not simply a type of data architecture but a new approach for designing modern data architectures by embracing organizational constructs as well as data-centric ones, data management, governance, etc. To understand how it works, we need to understand the four principles of data mesh: domain-driven design (DDD), data as a product, data access, and federated data governance. Confluent's free Data Mesh 101 course is a detailed video-based learning path for exploring data mesh concepts. The course covers in detail the four key principles of data mesh.
Data mesh solves a number of common problems. At the smaller scale, it addresses many of the issues seen with data pipelines, which often become brittle and problematic over time by creating their own webs and messy point-to-point kind of systems. It also addresses larger organizational issues, such as different departments in a company disagreeing on core facts of the business. In a data mesh, you’re less likely to have copies of facts. In both of these cases, a data mesh can bring much needed order to a system, resulting in a more mature, manageable, and evolvable data architecture.
The Definitive Guide to Building a Data Mesh with Event Streams blog post provides a summary of the foundational concepts and includes a data mesh prototype that you can build and run yourself.
The data mesh prototype is built on Confluent Cloud using event streams, ksqlDB, and the fully managed data catalog API. The prototype's GitHub repository has all the details.
Want to learn more about data mesh and how to use it? For questions, feedback, or to open a discussion on data mesh, sign up and engage with the friendly Confluent Community. For more resources like this, be sure to explore all of Confluent Developer and find complete courses, tutorials, examples, and more.