Security Recommendations

Before we go, I wanted to provide you with a checklist of the steps and decisions you need to make to secure your Apache Kafka Cluster. While this checklist won't cover all edge cases, it should serve as an outline of the steps that should be taken for most installations. The most important step to setting up a secure Kafka Cluster is education. Educate yourself on what your organization already uses and has set up for their other systems. Most likely you'll want to use the same systems and tools already being used so the security team doesn't have to provide you with a custom solution which will save everyone time and effort. This also helps down the road when you decide you want to make your Kafka Cluster available to more people, you don't have to wait around for security to be set up and verified. The next recommendation is to start with security in mind. Securing your Kafka System should be one of the success criteria for your cluster. Don't put it off to be configured later. Often making things secure at the end tends to break things which can be infuriating. Build security into the development environment before you even start producing and consuming data. it'll be ready to go once you're ready to publish it to production. Due to the wide spectrum of needs from different industries, it's impossible to give a do this and you'll never have to worry solution. I can however provide you with a checklist of things you need to discuss and have a plan for. I can also give you some recommendations that aren't required but will help start you out on the right foot. First, make sure you're using an encrypted file system and setting appropriate permissions for the users who need access. You'll want to make sure that all traffic to and from your cluster is encrypted using TLS. For situations where the data is highly sensitive, you should be using end to end encryption. In installations where you are using a cloud provider, such as AWS, Google Cloud, or Azure, we recommend using end to end encryption. Know and have a plan for the administration of ACLs. If not planned and managed, this can quickly become a headache and possibly introduce security holes into your cluster. You'll also wanna make sure you've set up a robust way of decommissioning accounts either in the event of a breach or in the case of an employee leaving your company. Another recommendation is to set up a key rotation policy. Security experts recommend having a policy to rotate your keys on a regular basis. If at all possible, you should look into having keys automatically rotated. You'll also wanna set up your broker certificates to dynamically update before they expire. While being a recommended security practice throughout the industry, it also prevents someone from getting access to all of your data. If you're using SASL, make sure that you've enabled reauthentication by setting connections.max.reauth.ms. This forces your connections to reestablish a connection and have their credentials verified on an ongoing basis. ZooKeeper should be segmented from devices on the network with only your brokers and administrative tools having access. All communications should also be encrypted using TLS. Audit logs are one of the best offensive measures you can take to verify and keep your system secure. Make sure you've configured and set up audit logging to track and verify configuration and access to your cluster. Last but not least, don't be afraid to play with an environment. There are numerous docker instances of fully set up and configured Kafka Clusters both from the community and from Confluent. Pick one or build one of your own and play around. Don't be afraid to mess things up and start over. The more time you spend configuring and testing, the better prepared and informed you'll be. While this is by no means an exhaustive list of everything you need to know or do to secure your Kafka installation, it should get you started. as has been mentioned throughout the course, you'll also want to check out the official Apache Kafka documentation and the free book, "Kafka: The Definitive Guide." If after watching this course you feel a little overwhelmed with all the security steps and configurations you need to administer, I highly recommend checking out Confluent Cloud, which provides the best cloud Kafka service with enterprise grade features, security, and zero ops burden. Most of the considerations we've talked about in this course have been addressed and make it simple for you to have a secure cluster that you can play with until you are ready to move things into production. Make sure you use the promo code with this course to get additional credits to try things out. Thanks and until next time.