Implementing a Data Mesh

Implementing a Data Mesh

This module covers the typical journey you might go through when implementing a data mesh in your organization. It won't cover the technical nitty-gritty details (code, APIs, etc.), but will rather give guidance into how a data mesh architecture works. The hands on modules after this one will walk you through the prototype setup in module 2.

You can view the four principles of the data mesh, covered in previous modules, as an evolution:

For example, you wouldn't start with Principle 3, data available everywhere, as a self service, rather, you'd start with Principle 1, data ownership by domain. You'd move from there to Principle 2, data as a product, and so on. Each principle generally features an increasing level of difficulty, but as you move through each of the data mesh principles, your capabilities will develop as well.

Data mesh has a concrete implementation, but it is more importantly a set of ideas. Reducing those ideas to practice by building out the mesh is a journey. So if you get started now, you're probably not going to have a data mesh next week. But there will come a point when you can reasonably say "I've built a data mesh." There may not be an obvious threshold, but you will recognize when you've reached that point.

Getting Started with Data Mesh - First Steps

To begin a data mesh, you first need to get the necessary management commitment. Then start with some concrete initial use cases: Ideally, things that are contained, simple, and owned by high-capability, forward-looking teams. You also want them to be visible, i.e., you want results that you can show to the business.

While data mesh is a valuable concept, it's not everything. It works in conjunction with other important systems such as microservices and domain-driven design, as we've mentioned. So those other methods are most likely going to need to be a part of your work, alongside and sometimes even orthogonal to data mesh. Basically, you should apply the data mesh concepts as you see fit to gain the maximum benefit for your company.

Steps to Implement a Data Mesh in Practice

• Centralize data in motion. Introduce a central event streaming platform; Kafka and Confluent Cloud are good solutions.
• Nominate data owners. You should have firm owners for the key datasets in your organization, and you want everyone to know who owns which dataset.
• Publish data on demand. You can store events in Kafka indefinitely, or they can be republished by data products on demand.
• Handle schema changes. Owners are going to publish schema information to the mesh (perhaps in the form of a wiki, or data extracted from the Confluent Cloud Schema Registry and transformed into an HTML document), and you need a process to deal with schema change approval.
• Secure event streams. You need a central authority to grant access to individual event streams. There are probably regulatory concerns here—perhaps even actual laws.
• Connect from any database. There are source and sink connectors available for many supported database types, and you should make sure that your desired connectors exist so that you can easily provision new output ports and input sources.
• Make a central user interface for discovery and registration of new event streams. This can be an application you create, or even a wiki. Ultimately, you're going to need to support searching for schemas for data of interest. You also need to support previewing event streams and requesting access to new event streams, and you need to support data lineage views.

Continue to the following modules to evaluate the data mesh prototype created in module 2, Hands On: Data Mesh Prototype Setup.