Apache Kafka 3.1 - Overview of Latest Features, Updates, and KIPs

Danica Fine:

Hi, I'm Danica Fine, developer advocate at Confluent. Welcome to another episode of Streaming Audio. This is a special edition where I'm announcing Apache Kafka 3.1 on behalf of the Kafka community. We've got great kits and updates in this release, but before we get to it, streaming audio is brought to you by Confluent Developer, that's developer.confluent.io, a website with everything you need to get started learning Apache Kafka. There are executable tutorials, a library of event-driven design patterns, video courses, and everything you'll need. For now, let's get to the Apache Kafka 3.1 highlights.

Danica Fine:

We’ve got a lot of great KIPs in this release, but we’ll stick to covering just a handful of the main ones from both Kafka Core and Kafka Streams. Let’s dive in!

Danica Fine:

First up, we have KIP 748 which brings us a couple of new broker count metrics – FencedBrokerCount and ActiveBrokerCount. These will be exposed by both the KRaft and ZooKeeper controllers, but the values will differ slightly depending on which you’re using, so keep that in mind. But the bottom line is that they both will respectively expose the number of active brokers in the cluster known by the controller and the number of fenced brokers known by the controller. Occasionally, a broker may be alive but unable to establish new connections, emit metrics, or emit logs due to failed DNS resolution. This condition can be hard to detect if the broker is not emitting metrics or logs, so, as you can imagine, these metrics are a great addition to the host of other broker metrics already available to you.

Danica Fine:

Next we have KIP 768 that extends SASL/OAUTHBEARER with support for OIDC. The industry is rallying around the OAUTH/OIDC framework for authorization and authentication so naturally it made sense to incorporate it here. The work in this KIP might sound familiar as the design and groundwork for supporting this was completed as part of KIP 255 – but that wasn’t quite a production-ready implementation. KIP 768 finally implements the necessary production-grade interfaces to allow for out of the box configuration by any Kafka user to connect to many popular external identity providers like Okta and Auth0.

Danica Fine:

Rounding out the Kafka Core KIPs is KIP 773 which provides some consistency in the way client metrics are named, specifically where millis and nanos are involved. A handful of metrics will be reintroduced with new names that reflect this. bufferpool-wait-time-total, io-waittime-total, and iotime-total will now include “ns” for nanoseconds. The metrics without “ns” in their name will be available for backwards compatibility only; they are marked deprecated and will be removed in the next release.

Danica Fine:

Now for Kafka Streams! KIP 763 and KIP 766 come together to make it possible to query state stores for ranges of values – but with open endpoints. This of course makes it possible to select a range greater than or less than a value. The functionality is available in both the ReadOnlyKeyValueStore and ReadOnlySessionStore.

Danica Fine:

Finally, KIP 775 allows you to use custom partitioners in your foreign key joins. Prior to this KIP, both the primary table and the foreign key table had to be partitioned using the default partitioner. To make this new functionality possible, the custom partitioners will be passed into a new TableJoined object which extends NamedOperation. Note that any foreign key join methods that take in NamedOperation will be deprecated moving forward; they will instead take in the TableJoined object.

Danica Fine:

Alright, that's all I have for now. As I said, there are quite a few other KIPs involved in this release. So as usual, I encourage you to head on over to our Confluent blog or take a look at the release notes to check them out in more detail. Can't wait to see what you build. That's the summary.

Danica Fine:

Thank you for listening to this episode. I hope this podcast was helpful to you. If you have any questions or would like to discuss, you could reach out to our community forum or Slack, both are linked in the show notes. If you're listening on Apple Podcast or other podcast platforms, please be sure to leave a review. We'd love to hear your feedback. If you're watching on YouTube, please do subscribe. We will notify you with updates you may be interested in. Thank you for your support. See you next time.