Get Started Free
‹ Back to courses
course: Apache Kafka® Security

Security Recommendations

5 min
dan-weston

Dan Weston

Senior Curriculum Developer

Security Recommendations

Before you go, we want to provide you with a couple of general recommendations as well as a security checklist. The checklist won't cover all use cases but it should serve as a useful outline.

Education

Learn the systems and tools that your organization already uses so that your security team doesn’t have to provide you with a custom solution. This will save setup time now and additional time in the future if you decide to make your Kafka cluster available to more teams in your organization.

Start with Security in Mind

Securing your Kafka cluster should be a key success metric of your project, rather than an afterthought. It is much easier to start from a secure standpoint in a development environment, rather than to try and add it retroactively to production.

Checklist

Due to the wide variety of Kafka use cases across industries, it’s impossible to suggest a “do this and you'll never have to worry” solution. However, what follows is a list of items to discuss and have a plan for, and some recommendations that aren’t required but that will help you to start out on the right foot.

Encrypt the filesystem

  • Use an encrypted filesystem and set appropriate ACLs.

Secure data in transit

  • Make sure all traffic to and from your cluster is encrypted using TLS. If your data is particularly sensitive, use end-to-end encryption.
  • End-to-end encryption is also recommended if you are using a cloud provider like AWS, Google Cloud, or Azure.

Set up a system for administering ACLs

  • ACLs can quickly become a headache and can introduce security holes if they aren't managed.
  • Also make sure you have a plan for decommissioning accounts, either in the event of a breach or in the case of a departing employee.

Rotate your keys

  • This should be done automatically, if possible.

Dynamically update certificates

  • Set your broker certificates to dynamically update before they expire, a standard industry practice that prevents someone from gaining access to all of your data.

Enable reauthentication

  • Set connections.max.reauth.ms to enable reauthentication if you are using SASL, which forces your connections to reestablish their connections and have their credentials verified on an ongoing basis.

Protect ZooKeeper

  • Only your brokers and administrative tools should have access to ZooKeeper, and it should be segmented from other devices on your network; all communication should also be encrypted with TLS.

Set up and monitor audit logs

  • Configure and set up audit logs to track and verify configuration and access to your cluster, as they are one of the best offensive measures you can take to verify and keep your system secure.

Play, tinker, break things

  • Finally, don't be afraid to play around in an environment. Use one of the many Docker-based, fully set up, and configured Kafka clusters from the community or from Confluent to experiment with, and then start over. The more time you spend tinkering, the better prepared you'll be.

This list is not exhaustive but should provide you with the means to get started. Make sure to review the official Kafka documentation as well as Kafka: The Definitive Guide for more details.

Additionally, keep in mind that managed Confluent Cloud removes much of the heavy lifting with respect to ops and security and provides the best cloud Kafka service with enterprise-grade features. If you do try out Confluent Cloud, make sure to use the promo code with this course to get additional credits.

Use the promo code 101SECURITY to get $25 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.

Security Recommendations

Before we go, I wanted to provide you with a checklist of the steps and decisions you need to make to secure your Apache Kafka Cluster. While this checklist won't cover all edge cases, it should serve as an outline of the steps that should be taken for most installations. The most important step to setting up a secure Kafka Cluster is education. Educate yourself on what your organization already uses and has set up for their other systems. Most likely you'll want to use the same systems and tools already being used so the security team doesn't have to provide you with a custom solution which will save everyone time and effort. This also helps down the road when you decide you want to make your Kafka Cluster available to more people, you don't have to wait around for security to be set up and verified. The next recommendation is to start with security in mind. Securing your Kafka System should be one of the success criteria for your cluster. Don't put it off to be configured later. Often making things secure at the end tends to break things which can be infuriating. Build security into the development environment before you even start producing and consuming data. it'll be ready to go once you're ready to publish it to production. Due to the wide spectrum of needs from different industries, it's impossible to give a do this and you'll never have to worry solution. I can however provide you with a checklist of things you need to discuss and have a plan for. I can also give you some recommendations that aren't required but will help start you out on the right foot. First, make sure you're using an encrypted file system and setting appropriate permissions for the users who need access. You'll want to make sure that all traffic to and from your cluster is encrypted using TLS. For situations where the data is highly sensitive, you should be using end to end encryption. In installations where you are using a cloud provider, such as AWS, Google Cloud, or Azure, we recommend using end to end encryption. Know and have a plan for the administration of ACLs. If not planned and managed, this can quickly become a headache and possibly introduce security holes into your cluster. You'll also wanna make sure you've set up a robust way of decommissioning accounts either in the event of a breach or in the case of an employee leaving your company. Another recommendation is to set up a key rotation policy. Security experts recommend having a policy to rotate your keys on a regular basis. If at all possible, you should look into having keys automatically rotated. You'll also wanna set up your broker certificates to dynamically update before they expire. While being a recommended security practice throughout the industry, it also prevents someone from getting access to all of your data. If you're using SASL, make sure that you've enabled reauthentication by setting connections.max.reauth.ms. This forces your connections to reestablish a connection and have their credentials verified on an ongoing basis. ZooKeeper should be segmented from devices on the network with only your brokers and administrative tools having access. All communications should also be encrypted using TLS. Audit logs are one of the best offensive measures you can take to verify and keep your system secure. Make sure you've configured and set up audit logging to track and verify configuration and access to your cluster. Last but not least, don't be afraid to play with an environment. There are numerous docker instances of fully set up and configured Kafka Clusters both from the community and from Confluent. Pick one or build one of your own and play around. Don't be afraid to mess things up and start over. The more time you spend configuring and testing, the better prepared and informed you'll be. While this is by no means an exhaustive list of everything you need to know or do to secure your Kafka installation, it should get you started. as has been mentioned throughout the course, you'll also want to check out the official Apache Kafka documentation and the free book, "Kafka: The Definitive Guide." If after watching this course you feel a little overwhelmed with all the security steps and configurations you need to administer, I highly recommend checking out Confluent Cloud, which provides the best cloud Kafka service with enterprise grade features, security, and zero ops burden. Most of the considerations we've talked about in this course have been addressed and make it simple for you to have a secure cluster that you can play with until you are ready to move things into production. Make sure you use the promo code with this course to get additional credits to try things out. Thanks and until next time.