Apache Kafka 3.2 - New Features & Improvements

Danica Fine: (00:00)

Welcome to another episode of Streaming Audio. I'm Danica Fine, Senior Developer Advocate at Confluent, and you're listening to a special episode where I get to do the honor of announcing the Apache Kafka 3.2 release on behalf of the Kafka community. There are so many great KIPs, highlights, and updates in this release. So, let's get to it.

Danica Fine: (00:24)

As usual, we'll be breaking this up into sections based on what the KIP pertains to. We'll cover updates around Kafka Core, Kafka Streams, and Kafka Connect. We'll kick things off with Kafka Core. First up, we have KIP-704, which allows for further communication between the controller and the brokers in the event that an unclean leader election has taken place. In this scenario, the controller sends messages to the brokers stating that an unclean leader election has happened, and that the topic partition leaders ought to recover their state. This helps to coordinate recovery and mitigate potential inconsistencies.

Danica Fine: (01:00)

KIP-764 introduces a new configuration parameter, socket.listen.backlog.size. Increasing this integer allows for a larger number of connections to be made and avoids filling up the acceptor backlog during leader election. Next, we have KIP-784, which made DescribeLogDirs response more consistent with other responses. With the change, an empty response no longer indicates in authorization error. This also opens the door for currently unknown error codes to be added in the future to further classify error responses. KIP-788, better handles network traffic by adding a new configuration to specify the number of network threads per listener. It's pretty common to have multiple types of listeners to find across Kafka brokers, but some of these listeners may have less traffic to handle than others. With KIP-788, the num.network.threads parameter has been altered and can now be prefixed with a specific listener name, allowing network threads to be configured on a per listener basis.

Danica Fine: (02:00)

With KIP-798 and KIP-810, Kafka-console-producer, everyone's favorite Kafka debugging and testing tool, has undergone a crucial facelift. Up until now, Kafka-console-producer was unable to produce messages with headers or null values. Currently scripting with Kafka-console-producer? No worries. These new features are handled with brand new flags, meaning that the existing functionality remains unchanged. The addition of KIP-800 means that we no longer need to guess the reason behind a consumer joining or leaving a consumer group. Clients log a reason when they join or leave a group. But it's difficult to collect this information from the broker side of things for further troubleshooting. With this change, the reason is propagated up to the brokers, making it easier to debug potential consumer group issues.

Danica Fine: (02:48)

For those of you excited about KRaft, KIP-801 introduces the first built-in authorizer that doesn't depend on ZooKeeper. In general, this new StandardAuthorizer does all of the same things that AclAuthorizer does for ZooKeeper-dependent clusters. StandardAuthorizer will be the default authorizer for KRaft-enabled clusters, but you're still free to specify another authorizer if you prefer. And wrapping up our Kafka core updates is KIP-814. This change makes metadata more consistent across a Kafka cluster in the event that a leader leaves and rejoins a group while static membership protocol is enabled. Before this change, a leader was allowed to be absent for a short time and rejoin the group while remaining the leader, but it was never informed that this happened. With this change, the functionality isn't affected. The leader can still leave, rejoin, and remain the leader, but it's now informed of its continued leadership.

Danica Fine: (03:42)

Next up we'll address a few Kafka Streams updates. KIP-708 increases fault tolerance of Kafka Streams by making it rack aware. Individual Kafka Streams clients can be tagged, for example, with our cloud region and cluster. Kafka Streams can then consider Stream clients' cloud region and cluster when distributing standby tasks. With KIP-805, we introduce RangeQuery, a new implementation of the Query interface provided by interactive queries version two. RangeQuery allows a state to be queried over a specific range. Expanding on this KIP is KIP-806, which brings WindowKeyQuery and WindowRangeQueryes class to interactive queries version two. Now you can access either a specific key or all keys within windowed state stores. The bottom line is that both KIP-805 and KIP-806 make it easier than ever to access information from underlying state stores through the new version of interactive queries.

Danica Fine: (04:37)

Our last Kafka Streams update is KIP-791. The StateStore context has been enhanced to include record metadata that may be accessed from state stores. This additional metadata will open the door for the implementation of stronger consistency semantics in the future.

Danica Fine: (04:52)

From the Kafka Connect side of things, our first update is KIP-769. Prior to this change, the Connect plug-ins endpoint printed only connectors. The API has now been extended to include a new connector's only flag, which when set to false will display all available plug-ins, not just the connectors. Using a plug-in output from the previous endpoint, you can also query the configuration of each plug-in. Up next is KIP-808, which supports new unix precisions in the timestamp converter single message transform. Using the new unix precision configuration parameter, you can easily specify your desired timestamp precision as you transform to and from unix timestamps.

Danica Fine: (05:34)

And finally KIP-779 adds the ability for connector source tasks to handle producer exceptions. Prior to this change, when a source task encountered a producer exception, it killed the entire connector. There wasn't really a way to properly recover. Worker source tasks will now utilize a getter method within the retry with tolerance operator and check to see how the exception should be handled either by skipping the record or failing the task.

Danica Fine: (05:59)

All right, those are the highlights from this latest Apache Kafka release. Thank you for listening to this episode, and I hope this podcast was helpful to you. If you have any questions or would like to discuss, you can reach out to our community forum or Slack, both are linked in the show notes. If you're listening on Apple Podcast or other podcast platforms, please be sure to leave a review. We'd love to hear your feedback. If you're watching on YouTube, please do subscribe. We'll notify you with updates that you might be interested in. Thank you again for your support and see you next time.