Hands On: Creating Your Own Data Mesh

This hands-on exercise will walk you through building and using a Data Mesh prototype application. This application is built as a web-server with Java Spring Boot using Confluent Cloud services, including the new Data Catalog API for data product management. The browser-based client is built using Elm and is viewable from your browser after you complete a few simple build steps. Let's get started. First, navigate to the data-mesh-demo GitHub repository where you can find the source code and documentation on the prototype. You can find the link in the module text below. In order to build and run the demo, there are some developer tool prerequisites you'll need installed on your system before proceeding. You'll need 'git' to clone the project, and also the 'make' tool to run the data mesh creation script. Additionally, you'll need the jq JSON processing tool. You can choose to run the data mesh prototype either by using a pre-built Docker image or by building and running from source yourself. By default, we will run the application using Docker so you'll need that. If you prefer to build and run from source, see the README in the repository for those instructions. The data mesh prototype automates the creation of Confluent Cloud resources including an environment, a basic cluster, topics, and a ksqlDB application. In order to run the script that provisions these resources, you will need the Confluent Command Line Interface installed on the system 'PATH'. The CLI can be installed using the provided installer script. You will also need a Confluent Cloud account to proceed. If you do not already have an account, you can sing up using the Confluent web console or by visiting confluent.cloud/signup New accounts are given $400 of free usage within the first 60 days. But keep in mind that this offer is subject to change over time, so you may see a different number. You also have the option of signing up for an account using the confluent CLI by running confluent cloud-signup and following the prompts as shown here. In addition to the new account usage, be sure to apply the promo code DATAMESH101 to your account, for an additional $101 of free Confluent Cloud usage. You can apply the promo code using the Cloud console under the billing section, or directly from the CLI. Finally, you will need a payment method on file to proceed with the demonstration. You can add a payment method using the cloud console, or again, with the CLI. All right, I know that was a lot of prep work to get started. In case you missed anything, be sure to take a look in the module notes below to see a complete list of everything you need to have ready before you move on to creating the data mesh prototype. With those housekeeping items in order, let's dive in. After cloning the source code repository, navigate into directory where the project was cloned. As mentioned earlier, the project uses the Confluent Cloud CLI to provision the Data Mesh resources on Confluent Cloud. If you didn't use the CLI for the signup process, you need to be sure your CLI is logged into your account before proceeding with the next steps. Be sure to use the --save option when you login. The '--save' flag simply allows your credentials to be saved locally to a file for future use and doing so, will prevent timeouts while the data mesh creation script is running. The project uses a Makefile to bootstrap the entire data mesh environment, including Confluent Cloud resources, a Java Spring Boot web server container, and a browser based client application that you can use to interact with the mesh. Kick off the mesh creation by running 'make data-mesh'. This script can take over 15 minutes to complete as it provisions various cloud resources. So while this command is running, you may continue with the next modules in the course and we'll pick back up where we explore the created data mesh.