Securing ZooKeeper

A system is only as secure as its weakest component. Zookeeper stores a lot of important cluster configuration and security information, including things such as a list of brokers that are currently members of the cluster, and Kafka ACLs, which are used for authorizing client requests to Kafka brokers. Zookeeper is also used to store consumer metadata, such as partition offset. With the introduction of its own native consensus implementation based on Raft, called KRaft, Kafka now supports Zookeeper list deployments. With KRaft, all of the metadata that was formerly stored in Zookeeper is now maintained in replicated internal topics. Now when you deploy Kafka, you can choose to run with either the traditional Zookeeper-based setup or KRaft. If you've chosen to run Kafka with Zookeeper, this module is for you. If you're using KRaft, you can skip this module. Decided? Okay, let's continue. As we mentioned in the beginning, in a Kafka cluster that includes Zookeeper, Zookeeper is used to store important metadata about your cluster and its clients, things such as ACLs and partition metadata. If you're securing using Kafka SASL or SCRAM provider for authorization, then Zookeeper is also being used to store encrypted versions of user passwords. This is all very important metadata. Any inappropriate access or manipulation of this metadata could seriously disrupt or compromise your cluster. If you're securing Kafka, then you should also be securing Zookeeper. If Zookeeper is left unsecured, then no matter how secure the rest of the system, it's still susceptible to being compromised. As with Kafka, there are two different ways of securing Zookeeper: SSL client authentication and SASL. Irrespective of the authentication mechanism you use, update your broker configuration with zookeeper.set.acl=true. This ensures that secure Zookeeper Access Control Lists, ACLs, are associated with the metadata in Zookeeper. These ACLs in turn ensure that the metadata can be read by everyone but modified only by the brokers. The exception here is sensitive metadata, such as SCRAM credentials, which cannot be read by everyone but only by the brokers by default. Zookeeper's SSL client authentication mechanism uses certificates to authenticate Zookeeper to brokers and the brokers to Zookeeper. There's one wrinkle with the setup, however, that distinguishes it from SSL client authentication in Kafka. Every broker, and in fact, every CLI tool, must use the same Distinguished Name when accessing Zookeeper. A certificate's Distinguished Name is the identifying information used to create the certificate and is part of the certificate. The reason Distinguished Name, or DN, must be the same for all brokers is because the DN is included in the Zookeeper ACL and Zookeeper only authorizes ACLs. Effectively, it only authorizes that Distinguished Name. Zookeeper will quite happily authenticate brokers with different Distinguished Names, but it will only authorize brokers with the same Distinguished Name. If broker A creates a piece of metadata in Zookeeper, broker B will only be authorized to modify it if it shares the same DN as broker A. A Distinguished Name typically includes the hostname of the entity to which it has been issued so as to allow the server, Zookeeper in this instance, to verify the hostname of the client making the request. If you've multiple brokers in your cluster running on different machines, then you'll have different hostnames. If you allow Distinguished Names to vary among broker certificates, you may find that brokers cannot access Zookeeper. The solution here is to either use a wildcard certificate, which includes a wildcard in the hostname, or a certificate with a single Distinguished Name plus a Subject Alternative Name, or SAN, containing a list of the actual hostnames of each of the brokers in the cluster. This single shared Distinguished Name will be used in the ACL, while the Subject Alternative Name allows Zookeeper to verify the actual hostname of the broker issuing a request. It's a simple point. Every client must use the same identity when being authorized by Zookeeper. It's just that the configuration and administration when using SSL client authentication require a bit of care and attention. Besides SSL client authentication, Zookeeper also supports SASL. As with Kafka, the main use case here is integration with an existing Kerberos server, such as Active Directory. If you use SASL, you should also ensure that connections from Kafka brokers to Zookeeper are encrypted using TLS. You can do this by setting ssl.clientAuth=none in the Zookeeper configuration. When set to none, Zookeeper allows clients to connect using a TLS encrypted connection without having to present their own certificate. With SASL and Kerberos, the problem of using the same identity for all brokers for ACL and authorization purposes is easily solved. All you have to do is ensure the Zookeeper client file on each broker is configured with the same principle. Which should you choose, SSL client authentication or SASL? SSL client authentication has some administrative overhead in terms of generating and installing certificates. You'll also need to periodically update this information before the certificates expire to prevent TLS handshake failures. Nonetheless, this is probably the easiest and the most popular option. SASL makes sense if you're already using Kerberos in your environment. If you go down this route, you'll need to integrate with your Kerberos server and ensure that these additional interactions are locked down. You can, in fact, use both. If you're using SSL client authentication in conjunction with SASL, then either the SSL identity or the SASL identity may be used to determine access and grant authorization, so all clients don't have to use the same Distinguished Name and therefore you can use hostnames in the Distinguished Name. While the measures we've discussed here will help secure Zookeeper, it is highly recommended that you also use network segmentation so that only your brokers and administrative tools have access to Zookeeper.