In this exercise, you will be setting up Confluent Cloud and Flink for use in the rest of the course. You'll also download the project template that contains the basic code you will need to get started.
The exercises come with some pre-existing code. You have two options for accessing it: Local or Gitpod.
Local is a good choice if you want an environment to support future projects.
If you prefer to have a disposable environment or get stuck setting it up locally, then Gitpod is a good choice.
For local development, you need a suitable Java development environment including.
NOTE: It is critical that you use Java 11 when working on these exercises. Flink does not currently support anything other than Java 8 or 11. Furthermore, the exercise code assumes you are using Java 11 and may use features that are unavailable in Java 8. Make sure that your $JAVA_HOME points to a compatible version.
To easily switch between Java versions, we recommend using something like SDKMAN.
Clone or download the exercise code from Github:
For Gitpod development, use the following link:
When you open Gitpod, it will open a series of tabs.
You will also have four terminal tabs:
You can use these terminals to execute portions of your application or run utility scripts.
You will still need to create a Confluent Cloud account and create a cluster.
Note: If you already have a Confluent Cloud account, you can skip ahead to Create a New Environment.
Watch your inbox for a confirmation email and follow the link to proceed.
You will be asked to create a cluster. Feel free to ignore this. We'll be creating a cluster in a later step.
If you prefer to work from the command line, you can install the Confluent CLI by following the instructions here.
Then, refer to the Command Line Reference sections that follow many of the instructions below.
We will create a cloud environment specifically for this course. This will ensure isolation and allow for easy cleanup.
WARNING: To ensure the integrity of your production environment, do not use a production environment for this course, and do not use the environment you create for anything other than the course. We strongly recommend using a new environment with new credentials to protect yourself but also to provide easy cleanup.
Name your environment building-flink-applications-in-java.
When offered a choice on which Stream Governance Package to use, select Essentials.
Select the cloud and region where you want to create your Schema Registry and Stream Catalog (i.e. where you will be storing the metadata).
You can do this from the command line by running:
confluent environment create building-flink-applications-in-java
Once your environment is created, you will need to make it the active environment.
If necessary, list the environments and locate the Id for the building-flink-applications-in-java environment.
confluent environment list
Next, set it as the active environment:
confluent environment use <environment id>
Next, we need to create a Kafka Cluster for the course.
If you prefer, you can do this from the command line interface by running (Adjust the cluster name, cloud, and region settings as appropriate):
confluent kafka cluster create <Your Cluster Name> --cloud gcp --region us-central1 --type basic
We will need an API Key to allow applications to access our cluster, so let's create one.
Create a key with Global access.
Note: For secure production environments, you would want to select Granular access and configure it more securely.
Download and save the key somewhere for future use.
If you prefer, you can do this from the command line interface by running (Adjust the cluster name and Id as appropriate):
confluent api-key create --resource <Your Cluster Id> --description <Your Cluster Name>-key -o json > <Your Cluster Name>-key.json
If you have chosen Gitpod as your development environment, then you can skip this step as Flink has already been installed.
If you are developing on your local machine, you will need to install Flink locally.
Inspect the install_flink.sh file which downloads and extracts a suitable version of Flink. You can execute the script as is or run the individual commands yourself.
Note: Future instructions in the course will be assuming that you have a Flink installation at learn-building-flink-applications-in-java/flink-<flink-version>. If you install Flink in an alternate location you may need to adapt the instructions.
Navigate to the Flink bin folder.
Run your Flink cluster:
And run three task managers by executing the following command 3 times:
Ensure Flink is running by navigating to http://localhost:8081.
This brings us to the end of this exercise.
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.