Introduction to Kafka Security

Security is a primary consideration for any system design and your use of Apache Kafka is no exception. When deciding how to secure your Kafka-based system, The degree of security you choose to apply can vary based on several factors: On your internal corporate policy, on any industry or regulatory requirements that govern your data processing capabilities, and on the environment in which you're deploying your solution. A development environment that doesn't contain sensitive data will often not need to be as secure as a production environment, for example. You can of course, dial the security up to 10 or 11 across all deployments and be done with it, but adding security to a system comes with a performance cost. Notably the CPU overhead of encrypting data can be significant when using a high throughput system like Apache Kafka, up to 30% in some cases. While Kafka makes many internal optimizations to reduce this cost, it is nonetheless noticeable. On the other hand, focus too much on performance and you run the risk of leaving some important part of the system poorly protected. Before we get into the details, let's look at how data flows through Kafka and the places where we need to consider the security aspects of your system. We receive a message from our producers. That message is sent to the leader broker, which writes the message to the local log file. The follower broker fetches the message from the leader and also writes its message to the log file. The leader broker updates the partition state in ZooKeeper, keeping the Insync replicas updated. Last, we have a consumer that receives the message from the broker. With each step, there's a security decision that needs to be made. First, the broker authenticates the client to ensure the message is really coming from the configured producer. Our producer should then verify it has a secure connection to the configured broker before sending any messages. All of our data should be encrypted so it can't be read in transit. In addition, all of our data stores should be encrypted as well for our data at rest. Before we commit the message to the log, the leader broker should verify that the producer has the correct access permission to write to the topic. The same goes for our consumer. Does it have the access it needs to read from the topic? We should also be recording these operations to an audit log, giving us an audit trail in case something unexpected happens or we need to verify our clusters behaving as it should. The mix of corporate, industry, and environment requirements particular to your scenario determine what security measures you ought to apply to your system. This course covers the tools Kafka provides to help you meet your specific security needs and the labs show you how to apply these tools and implement the security appropriate to your workload. By the end of this course, you'll have gained an understanding of which tools to use across the range of scenarios.