Exercise - Setting up Confluent Cloud (CC)

As part of this course on Data Engineering, we will be creating pipelines leveraging streaming technologies to perform analytics for a video game. The managed provider we will be using to host these technologies is Confluent Cloud.

This first exercise sets up the resources we will need in future ones. We'll create a Confluent Cloud account and environment, set up a Kafka cluster for our streaming data with an API key for reading and writing to topics, and provision a Flink compute pool to run Flink SQL queries.

Create an account on Confluent Cloud

This section will guide us through creating a Confluent Cloud account and adding promo codes to help offset the cost of the exercises.

Navigate to Confluent Cloud's signup page

• Sign up for a new account with your preferred login credentials (or login if you already have an account) at Confluent Cloud's Signup page
• Click the verify link in your email. Don't worry about creating a new cluster, we'll do that shortly. Feel free to navigate to the main page at Confluent Cloud to play around with the UI. Stay logged in, because we'll add the promo codes in the next step.

Add promo codes

• On the top right, click on the menu icon, and select Billing & payment
• Click on the Payment details & contacts tab
• At the very bottom, add the following promo codes

DATAENGINEERING
CONFLUENTDEV1

Clone the GitHub repo

We'll need to clone the GitHub repository that contains all the code and configuration files for the exercises:

Clone it and move into the folder:

git clone https://github.com/confluentinc/learn-data-engineering-exercises.git
cd learn-data-engineering-exercises

This repository contains code and configuration files that we will use throughout the rest of the course. Ensure you are in this folder when running all the exercise commands.

Install the Confluent CLI and quickstart plugin

There is a plugin/quickstart that will make it easier to install all of our resources instead of running the individual commands.

Install the CLI

Bring up a command line terminal and install the Confluent CLI. Instructions can be found on the Confluent CLI documentation page. If you are on a Mac, you can install it via brew:

brew install confluentinc/tap/cli

Save your login credentials

To avoid logging in each time, save your credentials:

confluent login --save

Install the quickstart plugin

confluent plugin install confluent-quickstart

Create the Kafka environment, cluster, and Flink compute pool

Now we'll use the quickstart plugin to create all the necessary infrastructure with a single command. This will set up our environment, Kafka cluster, and Flink compute pool.

Create the Kafka and Flink resources

confluent quickstart \
   --environment-name data_engineering \
   --kafka-cluster-name exercises \
   --cloud aws \
   --region us-west-2 \
   --create-kafka-key \
   --kafka-librdkafka-properties-file ./jr/kafka/config.properties \
   --compute-pool-name data_engineering \
   --max-cfu 10 \
   --create-tableflow-key \
   --tableflow-properties-file ./tableflow.properties

Note: Feel free to alter the region to something more appropriate for your location. However, make sure that if you change it, you use the same settings throughout the course. You can use the CLI to list the currently available AWS regions:

confluent kafka region list --cloud aws

The same command is also available as the create_environment.sh script in the repository. The script uses us-west-2, so if you choose a different region, edit the --region value in the script before running it.

This quickstart creates a new environment, a new Kafka cluster, as well as a Flink compute pool. The Kafka bootstrap server, along with the access and secret keys, are stored in ./jr/kafka/config.properties. The Tableflow API key, secret, catalog endpoint, and region are stored in ./tableflow.properties; we will use them in the Tableflow exercise.

Confirm your assets have been created

confluent environment list
confluent kafka cluster list
confluent flink compute-pool list

Finish

This brings us to the end of the exercise. We now have a fully functioning Kafka and Flink environment.

Make sure to hold on to the environment we created, and keep the Kafka key in a safe place because we will need them for the remaining exercises.

In the next exercise we will populate Kafka with data from our video game.