Hands On: Setting Up Encryption

In this hands on, I'll show you how to put a lot of things that we've talked about so far into practice. First, we'll set up our Kafka environment to use SSL and TLS to encrypt our data in motion, creating certificates and then configuring our brokers to also use SSL. We'll then configure the Kafka client to encrypt the data in transit using SSL and then require SSL for our broker traffic. The first thing we need to do is get our environment set up. If you remember from the first exercise, I've already cloned the learn Kafka courses and placed it as my working directory. Don't forget to go into the fund-kafka-security folder as well. I've also set the same working directory inside of Visual Studio Code. Before we start our Kafka instance, there are a few changes that we need to make to get things set up. First, let's take a look at the Docker Compose file that has the instructions for Docker, but also for our server configuration parameters. We'll go into Visual Studio Code and click on Docker Compose. This environment has three instances of ZooKeeper as well as three brokers. If I scroll down to look at my brokers, you'll notice that I have two listeners configured, the default plain text listener and the internal broker listener. Remember, both of these send data as plain text. You can also see that the advertised listeners have been configured on the next line. Let's go ahead and comment both of these lines out and remove the comment from the next two lines adding in the SSL listener. We'll want to do that for each broker. You'll also notice that we have Kafka Listener Security Protocol Map, which already has SSL as a connection as well. This is the property that determines the communication protocol used by the listeners. We'll go ahead and save that file and go over to our command line to start creating the certificates. Now only to create the certification authority key and the certificate by running the following command in the terminal. In this exercise, we're using a certificate that is self-signed. As mentioned in the previous modules, if you're using a production environment, we recommend using a trusted certificate authority rather than a self-signed certificate. I'll paste in the command. Notice I've created a configuration file with a lot of the parameters, so that I don't have to manually enter them in. Here we're creating a new key and certificate that is valid for 365 days. It uses the RSA 248 encryption and uses the values that we've stored in the configuration file on the machine. Feel free to take a look at that file if you're interested in the parameters used as it's created. You should see an output similar to the one on screen. Now, we need to convert those files over to a .PEM file. We'll go ahead and create a new server key and certificate by running the following command and then sign the certificate with the certificate authority. Then we'll need to convert the server certificate over to the PKCS 12 format. Next we'll create the broker keystore and import the certificate. Okay, we see that it was successfully imported. Now we need to verify the broker keystore. And your output should look similar to what you see on the screen. Last but not least, we'll save the credentials so that we can use them later. Now, if you've noticed we've only done this for our kafka-one broker. We could go through and do it for the other two, or I've created a handy script that'll go ahead and finish building it out for both Kafka two and Kafka three. So we'll run that command and wait for it to finish. There we go. If we talk about the saved credentials, they're needed for the broker ssl.keystore.credentials and ssl.key.credentials broker configuration parameters. These parameters are set for our lab environment brokers in the environment section of the Docker Compose file. So let's go ahead and take a look at that. So this is on Kafka three broker, but you can see both the keystore credentials that are saved right there. Now let's go back and start up our Docker instance by running Docker Compose up. We'll make sure everything is running by running docker-compose ps. We can see our instances are running. Now I'll open an SSL connection with our Kafka one broker to verify that things are working. And you can see I'm connected. I can do the same thing with Kafka two. And last but not least Kafka three. Congratulations, we've successfully added an SSL listener to your broker, created a broker keystore, imported the CA to your broker keystore, and configured their SSL properties. In the next video, we'll create the Kafka client trust store and import the CA, configure the Kafka client to encrypt the data in transit, and then use SSL for client to broker traffic.