Hands On: Requiring Encryption for Broker Traffic
Dan Weston
Senior Curriculum Developer
Hands On: Requiring Encryption for Broker Traffic
In the previous exercise, we enabled SSL on our Kafka brokers. Now we'll take it even further by creating the Kafka client truststore and importing the CA, configuring the Kafka client to encrypt data in transit using SSL, and requiring SSL for client-broker traffic.
- If you didn't complete the previous exercise, you can easily set up your environment by running the mod2-init.sh inside the scripts folder:
./home/training/learn-kafka-courses/fund-kafka-security/scripts/mod2-init.shThis will load in the correct docker-compose file as well as create the certificates and save all the credentials.
- Start the Docker containers by changing to the fund-kafka-security directory and running
docker-compose:
cd fund-kafka-security
docker-compose up -d- The first thing we are going to do is create a new topic:
kafka-topics \
--bootstrap-server localhost:19092 \
--create \
--topic test-topic \
--replica-assignment 101:102:103- Now we need to monitor our network traffic while also producing so we can see and verify that our connection is encrypted. First, we'll set up the monitoring by downloading and running a netshoot Docker image:
docker run --rm -it --net container:kafka-1 \
nicolaka/netshoot \
tcpdump -c 40 -X port 19092 or port 19093Once that image has been downloaded and started you should see an output similar to the one in the video.
- Next, we'll open a new terminal window and run our console producer:
kafka-console-producer \
--bootstrap-server localhost:19092 \
--topic test-topicThen produce a message:
Kafka Rocks!Exit the console producer by typing Ctrl-D.
If we head back over to our monitoring terminal we can clearly read both the topic that we wrote to as well as the message "Kafka Rocks" in plaintext.
- Let's go back to the terminal where we were producing our messages and create the client truststore and import the CA certificate:
keytool -keystore /home/training/learn-kafka-courses/fund-kafka-security/client-creds/kafka.client.truststore.pkcs12 \
-alias CARoot \
-import \
-file /home/training/learn-kafka-courses/fund-kafka-security/ca.crt \
-storepass confluent \
-noprompt \
-storetype PKCS12- This time when we run the console producer, we'll use the SSL port to produce the messages:
kafka-console-producer \
--bootstrap-server localhost:19093 \
--topic test-topicAlmost immediately we start to see errors. Any idea why? Consider the command above. While we provided the SSL port, we also needed to provide the configuration settings. We can see what these are by looking at the client-ssl.properties file:
cat /home/training/learn-kafka-courses/fund-kafka-security/client-creds/client-ssl.propertiesLet's try that command again, but this time provide the configuration file.
First, we'll start our monitoring again:
docker run --rm -it --net container:kafka-1 \
nicolaka/netshoot \
tcpdump -c 40 -X port 19092 or port 19093Then we'll run the same command, but with the location of the client-ssl.properties file:
kafka-console-producer \
--bootstrap-server localhost:19093 \
--topic test-topic \
--producer.config /home/training/learn-kafka-courses/fund-kafka-security/client-creds/client-ssl.propertiesFantastic, no error messages. Let's send a message:
Kafka Rocks!Then close the producer by typing Ctrl-D.
This time when we look at the output we can see that the message was encrypted and is no longer in plaintext.
- Now that we know that the encryption is working, let's disable the plaintext listener. Open the docker-compose.yml file and scroll to the kafka-1 broker environment. We are looking for the KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS lines.
Go ahead and delete the PLAINTEXT sections from the line, and then do it for our other two brokers as well.
You can also delete it from the KAFKA_LISTENER_SECURITY_PROTOCOL_MAP section.
- Now recreate our brokers by typing:
docker-compose up -d- After things have come back up, we'll go ahead and try to retrieve the messages we sent to our test-topic:
kafka-console-consumer \
--bootstrap-server localhost:19092 \
--topic test-topic \
--from-beginningAfter you see the errors close the connection with Ctrl-C.
Similar to what we previously saw, the consumer was attempting to use the plaintext listener to retrieve the messages.
- If we now try to retrieve the messages from the SSL port and provide the configuration file we'll actually see our messages:
kafka-console-consumer \
--bootstrap-server localhost:19093 \
--topic test-topic \
--consumer.config /home/training/learn-kafka-courses/fund-kafka-security/client-creds/client-ssl.properties \
--from-beginningPress Ctrl-C to stop the consumer.
Congratulations! You successfully configured Kafka clients to encrypt traffic between themselves and your Kafka brokers (with plaintext listeners disabled).
Use the promo code 101SECURITY & CONFLUENTDEV1 to get $25 of free Confluent Cloud usage and skip credit card entry.
Be the first to get updates and new content
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.
Hands On: Requiring Encryption for Broker Traffic
In the previous hands-on, we enabled SSL on the Kafka brokers. Now we'll take it even further by creating the Kafka client trust store and importing the CA and configure the Kafka client to encrypt data in transit using SSL. If you didn't complete the previous exercise and you're just joining us in this video, you can easily set up your environment by running the following script. Now, since I'm just continuing on, there's no need for me to run that script. That script will load in the correct Docker compose file as well as the changes that we've made, create the certificates and save all the credentials for us. We'll go ahead and start by bringing up our Docker container. The first thing we need to do is create a new topic to write our data into. We'll call this test-topic and then we'll set up the replica assignment as well. Now we're gonna go ahead and set up a monitoring for our network traffic so that we can see the data as it's transferred to the topic. To do this, we'll download and run the netshoot Docker container. After it's finished downloading, it'll go ahead and start that container for us to look at. In the meantime, as that's coming up, I'm gonna go ahead and open up a new tab and start our console producer. Now I can go ahead and write a message and head back to our monitoring tab. Now, if we look at this closer, you can see that we can see that we were running the console-producer-test and we can see our message right there, Kafka rocks! So even though we've already set up SSL on our cluster, it's not using it for our producer messages. Okay, let's go ahead and stop monitoring and also stop our console producer. Okay, so now in the terminal tab where we were producing our messages, let's create the client trust store and import the CA certificate. This time as we run the console producer, you can see that we're actually using the port that's configured for SSL. And almost immediately, we start receiving errors. While we provided the SSL port, we also need to provide the configuration settings so that our console producer knows how to communicate. We can see what these are by looking at the properties file. This time, let's go ahead and start our monitoring one more time and notice we're listening on the SSL port, come back over to our console producer tab and provide the producer config. Fantastic, no error messages. Let's go ahead and send a message and pop back over to look at our monitoring terminal. And it looks like we were too slow, so we'll go ahead and run it again, come back over and type in a new message to send. Look over in the monitoring side and you'll notice everything looks very different than before. As you can see, everything is being encrypted. Now that we know that encryption is working, let's go ahead and disable the plaintext listener. We'll open up the Docker compose file and scroll to the kafka-1 broker environment. We're looking for the lines Kafka listeners and Kafka advertise listeners. We'll go ahead and delete the plaintext sections and then scroll to kafka-2 and do the same and then kafka-3 last, and then save the file. Now, popping back over to terminal, it looks like we forgot to close down our monitoring. And I bet that means we also forgot to close down our producing, so we'll go ahead and close down both of them and then restart our Docker image. After things have come back up, we'll go ahead and try to retrieve the messages we sent to our test-topic using the plaintext ports. And we get errors almost immediately. This is fantastic. This is exactly what we wanna see. The port is no longer listening and the listener is inactive. Similar to what we previously saw, the consumer was attempting to use the plaintext listener to retrieve the messages. If we now try to retrieve the messages from the SSL port and provide the configuration file, we'll actually see our messages. I'll hit Control + C to stop the consumer. Congratulations, you successfully configured Kafka clients to encrypt traffic between themselves and your Kafka brokers and disabled the plaintext listeners.