Get Started Free
January 24, 2022 | Episode 196

Apache Kafka 3.1 - Overview of Latest Features, Updates, and KIPs

  • Transcript
  • Notes

Danica Fine:

Hi, I'm Danica Fine, developer advocate at Confluent. Welcome to another episode of Streaming Audio. This is a special edition where I'm announcing Apache Kafka 3.1 on behalf of the Kafka community. We've got great kits and updates in this release, but before we get to it, streaming audio is brought to you by Confluent Developer, that's, a website with everything you need to get started learning Apache Kafka. There are executable tutorials, a library of event-driven design patterns, video courses, and everything you'll need. For now, let's get to the Apache Kafka 3.1 highlights.

Danica Fine:

We’ve got a lot of great KIPs in this release, but we’ll stick to covering just a handful of the main ones from both Kafka Core and Kafka Streams. Let’s dive in!

Danica Fine:

First up, we have KIP 748 which brings us a couple of new broker count metrics – FencedBrokerCount and ActiveBrokerCount. These will be exposed by both the KRaft and ZooKeeper controllers, but the values will differ slightly depending on which you’re using, so keep that in mind. But the bottom line is that they both will respectively expose the number of active brokers in the cluster known by the controller and the number of fenced brokers known by the controller. Occasionally, a broker may be alive but unable to establish new connections, emit metrics, or emit logs due to failed DNS resolution. This condition can be hard to detect if the broker is not emitting metrics or logs, so, as you can imagine, these metrics are a great addition to the host of other broker metrics already available to you.

Danica Fine:

Next we have KIP 768 that extends SASL/OAUTHBEARER with support for OIDC. The industry is rallying around the OAUTH/OIDC framework for authorization and authentication so naturally it made sense to incorporate it here. The work in this KIP might sound familiar as the design and groundwork for supporting this was completed as part of KIP 255 – but that wasn’t quite a production-ready implementation. KIP 768 finally implements the necessary production-grade interfaces to allow for out of the box configuration by any Kafka user to connect to many popular external identity providers like Okta and Auth0.

Danica Fine:

Rounding out the Kafka Core KIPs is KIP 773 which provides some consistency in the way client metrics are named, specifically where millis and nanos are involved. A handful of metrics will be reintroduced with new names that reflect this. bufferpool-wait-time-total, io-waittime-total, and iotime-total will now include “ns” for nanoseconds. The metrics without “ns” in their name will be available for backwards compatibility only; they are marked deprecated and will be removed in the next release.

Danica Fine:

Now for Kafka Streams! KIP 763 and KIP 766 come together to make it possible to query state stores for ranges of values – but with open endpoints. This of course makes it possible to select a range greater than or less than a value. The functionality is available in both the ReadOnlyKeyValueStore and ReadOnlySessionStore.

Danica Fine:

Finally, KIP 775 allows you to use custom partitioners in your foreign key joins. Prior to this KIP, both the primary table and the foreign key table had to be partitioned using the default partitioner. To make this new functionality possible, the custom partitioners will be passed into a new TableJoined object which extends NamedOperation. Note that any foreign key join methods that take in NamedOperation will be deprecated moving forward; they will instead take in the TableJoined object.

Danica Fine:

Alright, that's all I have for now. As I said, there are quite a few other KIPs involved in this release. So as usual, I encourage you to head on over to our Confluent blog or take a look at the release notes to check them out in more detail. Can't wait to see what you build. That's the summary.

Danica Fine:

Thank you for listening to this episode. I hope this podcast was helpful to you. If you have any questions or would like to discuss, you could reach out to our community forum or Slack, both are linked in the show notes. If you're listening on Apple Podcast or other podcast platforms, please be sure to leave a review. We'd love to hear your feedback. If you're watching on YouTube, please do subscribe. We will notify you with updates you may be interested in. Thank you for your support. See you next time.

Apache Kafka® 3.1 is here with exciting new features and improvements! On behalf of the Kafka community, Danica Fine (Senior Developer Advocate, Confluent) shares release highlights that you won’t want to miss, including foreign-key joins in Kafka Streams and improvements that will provide consistency for Kafka latency metrics. 

KAFKA-13439 deprecates the eager protocol, which has been the default since Kafka 2.4—it’s advised to upgrade your applications to the cooperative protocol as the eager protocol will no longer be supported in future releases. Previously, foreign-key joins in Kafka Streams only worked if both primary and foreign-key tables were joined. This release adds support for foreign-key joins on tables with custom partitioners, which will be passed in as part of a new `TableJoined` object, comparable to the existing `Joined` and `StreamJoined` objects. 

With the goal of making Kafka more intuitive, KIP-773 enhances naming consistency for three new client metrics with millis and nanos. For example, `io-waittime-total` is reintroduced as `io-wait-time-ns-total`. The previously introduced metrics without ns will be deprecated but available for backward compatibility. KIP-768 continues the work started in KIP-255 to implement the necessary interfaces for a production-grade way to connect to an OpenID identity provider for authentication and token retrieval. This update provides an out-of-the-box implementation of an `AuthenticateCallbackHandler` that can be used to communicate with OAuth/OIDC. 

Additionally, this Kafka release introduces two new metrics for active brokers specifically, `ActiveBrokerCount` and `FenceBrokerCount`. These two metrics expose the number of active brokers in the cluster known by the controller and the number of fenced brokers known by the controller. 

Tune in to learn more about the Apache Kafka 3.1 release! 

Continue Listening

Episode 197January 27, 2022 | 31 min

Expanding Apache Kafka Multi-Tenancy for Cloud-Native Systems ft. Anna Povzner and Anastasia Vela

In an effort to make Apache Kafka cloud native, Anna Povzener (Principal Engineer, Confluent) and Anastasia Vela (Software Engineer I, Confluent) have been working to expand multi-tenancy to cloud-native systems with automated capacity planning and scaling in Confluent Cloud. They explain how cloud-native data systems are different from legacy databases and share the technical requirements needed to create multi-tenancy for managed Kafka as a service.

Episode 198February 1, 2022 | 30 min

Intro to Event Sourcing with Apache Kafka ft. Anna McDonald

What is event sourcing, and how does it work? Event sourcing is often used interchangeably with event-driven architecture and event stream processing. However, Anna McDonald (Principal Customer Success Technical Architect, Confluent) explains it's a specific category of its own—an event streaming pattern.

Episode 199February 3, 2022 | 6 min

On to the Next Chapter ft. Tim Berglund

After nearly 200 podcast episodes of Streaming Audio, Tim Berglund bids farewell in his last episode as host of the show. Tim reflects on the many great memories with guests who have appeared on the segment—and each for its own reasons. He has covered a wide variety of topics, ranging from Apache Kafka fundamentals, microservices, event stream processing, use cases, to cloud-native Kafka, data mesh, and more.

Got questions?

If there's something you want to know about Apache Kafka, Confluent or event streaming, please send us an email with your question and we'll hope to answer it on the next episode of Ask Confluent.

Email Us

Never miss an episode!

Confluent Cloud is a fully managed Apache Kafka service available on all three major clouds. Try it for free today.

Try it for free