Hands On: Setup the Exercise Environment for Confluent Cloud and Python

Confluent Cloud Setup

Throughout this course, we’ll introduce you to developing Apache Kafka event streaming apps with Python through hands-on exercises that will have you produce data to and consume data from Confluent Cloud. This exercise will get you set up for the exercises that follow.

Sign up for Confluent Cloud

Note: If you already have a Confluent Cloud account, you can skip this section and proceed to step 8.

If you haven’t already signed up for Confluent Cloud, sign up now so when your first exercise asks you to log in, you are ready to do so.

1. Browse to the sign-up page: https://www.confluent.io/confluent-cloud/tryfree/
2. Fill in your contact information and a password. Be sure to remember these details as you will need them to log in to Confluent Cloud.
3. Click the Start Free button and wait for a verification email.
4. The link in the email will lead you to a screen where you will be prompted to complete a short survey. After doing so, click Continue or alternatively just click Skip.
5. When the information dialog appears that informs you of the promotional credit you can use to become familiar with Confluent Cloud, click Get started to continue with these set up steps.
6. When the next screen appears that prompts you to set up your first cluster, click I'll do it later.
7. Continue to step 9.

Login to the Confluent Cloud Console

8. Open URL https://confluent.cloud and log in to the Confluent Cloud console.
9. Navigate to the environments page and click Add cloud environment.
10. Name the new environment learn-kafka-python and click Create.

Enable Streams Governance

Next, you will be prompted to enable one of the available Streams Governance Packages. You need to do so since the course exercises utilize Schema Registry which these packages include.

11. Click Begin configuration for the Essentials package.
12. Select a provider and region and click Enable.

Create a Cluster in the learn-kafka-python Environment

13. Click Create cluster on my own.
14. Click Begin configuration for the Basic cluster.
15. Select a provider and region and click Continue.
16. Assign a name of kafka-python and click Launch cluster.

FREE ACCESS: Basic clusters used in the context of this exercise won't incur much cost, and the amount of free usage that you receive along with the promo code PYTHONKAFKA101 for $25 of free Confluent Cloud usage will be more than enough to cover it. You can also use the promo code CONFLUENTDEV1 to delay entering a credit card for 30 days.

Create and Download Python Client Configuration Properties

Next, you will create the configuration properties needed to connect to your cluster.

17. Navigate to the Cluster Overview Dashboard and for the kafka-python cluster.

18. Click on the Python tile in the Connect to your systems section.

19. Create a cluster API key and secret by clicking Create Kafka cluster API key.

20. Review the key and the secret in the pop-up and click Download and continue.

    This will download the key and secret to the local machine Downloads directory.

21. To create the Schema Registry API key and secret, repeat the process by clicking Create Schema Registry API key.

22. Review the sample Python client configuration settings. The API keys and secrets should be populated in the properties in the middle of the page. If they are not visible, select the Show API keys option.

23. Click the Copy button.

Setup Project

Create Python Dictionary

24. Create a file named python.properties located in the ~/.confluent directory.

    Note: The .confluent directory is the default location for Confluent Platform related configuration files.

25. Edit python.properties and paste the previously copied Python client Confluent Cloud properties.

26. Save python.properties but leave it open in your IDE.

27. Create a config directory named kafka-python for the course exercises and cd into it.

28. Create a file named config.py that will contain a Python dictionary.

29. Add the following lines to config.py:

    config = {
         'bootstrap.servers': '<bootstrap-server-endpoint>',     
         'security.protocol': 'SASL_SSL',
         'sasl.mechanisms': 'PLAIN',
         'sasl.username': '<CLUSTER_API_KEY>', 
         'sasl.password': '<CLUSTER_API_SECRET>'}

30. Update the values for bootstrap.servers, sasl.username, and sasl.password to the values contained in python.properties.

    We’ll add more to config.py in later exercises.

Install Python 3.X and Related Prerequisites on Local Machine

The course exercises depend upon Python 3.X and several related packages being installed. The steps in this prerequisites section satisfy this dependency. Complete the steps as needed on your local machine.

Note: These steps correspond to installing these packages on Ubuntu 20.04. Adjust the steps as needed to suit your local environment.

31. First update all the packages in your local machine:

    sudo apt update

32. If any of the system packages on your local machine need to be upgraded, run the following command to do so:

    sudo apt upgrade

33. Install python3:

    sudo apt install python3

34. Install pip3:

    sudo apt install python3-pip

35. Identify default python version:

    python --version

    The version shown in the command response should be similar to Python 3.X.X. If a non-Python 3.X version is returned, you will need to either update your alias for the python command or explicitly run the python3 and pip3 commands during the exercises for this course.

36. Install virtualenv:

    pip install virtualenv

Create a New Virtual Environment with virtualenv

37. Create a virtual environment:

    virtualenv kafka-env

38. Activate the virtual environment:

    source kafka-env/bin/activate

Install Confluent Kafka

39. Run the following command:

    pip install confluent-kafka

You’re now ready for the upcoming exercises.