Apache Kafka 4.3.0 Release is Here

The Apache Software Foundation recently released Apache Kafka® 4.3. This release contains many new features and improvements. With 25 KIPs and over 600 commits since 4.2.0, this release introduces many new features, improvements and bug fixes to all the components.

This release includes:

• KIP-1271 & KIP-1285: Header-aware State Stores in Kafka Streams
  • Kafka record headers are often used to carry metadata - think correlation IDs, tracing information, or feature flags. Before this release, Kafka Streams did not persist record headers in state stores - only the serialized key and value were stored. This means downstream processors could not rely on header metadata. As of this release, record headers are a first-class citizen in state stores - with a rolling-upgrade path from existing state stores.
• KIP-1240: Production Controls for Share Groups
  • This KIP introduces new configurations to provide more granular, independent control of share groups. For instance, an upper (group.share.max.delivery.count.limit) and lower (group.share.min.delivery.count.limit) bound can be established on the Kafka cluster for delivery count. Then a share group can set its own delivery limit to a value within those broker-defined bounds (share.delivery.count.limit). Another broker configuration sets a limit for the number of acquired records per share partition (group.share.max.partition.max.record.locks). The share group can then set its own limit (share.partition.max.record.locks) to a value less than the broker prescribed limit. The AcknowledgementType.RENEW can be enabled or disabled for a share group (share.renew.acknowledge.enable); enabled is the default value.
• KIP-1066: Broker Cordoning
  • The concept of “cordoning” is borrowed from the Kubernetes world. In Kafka, cordoning a log directory excludes it from receiving new partition allocations. If all log directories of a broker are cordoned, then the broker itself is effectively cordoned. Via the broker configuration cordoned.log.dirs, an operator can exclude directories from new partition writes while still servicing requests to existing partitions. This could be useful in maintenance scenarios - i.e. decommissioning a broker or removing a disk volume.
• KIP-1270: GlobalKTable Error Handling
  • Prior to this release, the ProcessingExceptionHandler in Kafka Streams applied only to “regular” streams tasks. This means an error in a GlobalKTableTask could potentially throw an Exception that crashed the application. As of 4.3, the ProcessingExceptionHandler be enabled (processing.exception.handler.global.enabled) to gracefully shut down the application. This will become the default behavior in the Kafka 5.0 release.

This release also has an eye on the future of the Kafka ecosystem, will some key deprecations and removals scheduled for version 5.0 and beyond:

• KIP-1274: Using the classic rebalance protocol in Kafka consumer configuration will log a message with an upgrade recommendation. As of Kafka 5.0, consumer will become the default protocol - as introduced in KIP-848. And in Kafka 6.0, the classic protocol will be removed from the codebase. Please start migrating ASAP!
• KIP-1244: Support for Kafka Streams Scala DSL will be dropped in AK 5.0. Please use the Java DSL instead.
• KIP-1237: The broker configuration group.coordinator.rebalance.protocols is slated for removal in AK 5.0.
• KIP-1211: This KIP aligns the behavior of num.partitions and default.replication.factor for topic creation, starting the process of moving configuration used in topic creation to the controller.

Read our release blog to learn more about the innovations included in Apache Kafka 4.3!

What’s New in Confluent Cloud

Whether you're bringing stream processing into SQL-based workflows, putting best practices into AI coding agents, or powering event-driven AI with enterprise-grade reliability, your data streaming platform should do the heavy lifting for you.

We’re excited to share Confluent Cloud’s Current London launches, including the dbt adapter for Confluent Cloud for Apache Flink, a managed MCP server, and more Confluent Intelligence releases.

• dbt adapter for CC Flink SQL: Free, open source plugin that enables data engineering teams to define streaming pipelines as dbt models, test them, generate documentation, and deploy streaming pipelines to Flink compute pools using the same dbt workflow they’re familiar with.
• Materialized Tables: Persistent, database-like objects that automate offset bookkeeping and job orchestration through a single SQL statement. Simplifies managing the life cycle of streaming pipelines.
• Managed MCP Server and Confluent Agent Skills: A fully managed MCP server for CC with a complementary catalog of Agent Skills, plus expanded support for the open source, local MCP server to move seamlessly between local and cloud development.

New Confluent Intelligence updates: 

• Real-Time Context Engine : Continuously serve fresh, trustworthy context via MCP to any AI agent or application anywhere.
• Agent Management Console: Create and operate Streaming Agents at scale with Agent Management Console, a centralized, UI-driven control plane.
• Streaming Agents: Monitor live business signals to take autonomous action with event-driven streaming agents built directly into Flink and Kafka pipelines.
• KCP: Automate the final mile of Apache Kafka migrations with KCP, which now orchestrates client migration in a seamless, safe, and rollback-friendly manner.
• Schema IDs in Kafka headers: Schematize existing topics in minutes with zero downtime, making it easier to create structured, governed events that can be reused across AI and ML workloads.

Learn More:

• Read the latest launch blogs: Q2 Confluent Cloud launch blog, Q2 Confluent Intelligence updates, Confluent's MCP Server & Agent Skills, CPC Benchmark and Multitenancy
• Check out the demos Real Time Context Engine, dbt adapter and Materialized Tables and KCP
• Missed the Keynote? Watch the Current London Keynote here

Data Streaming Resources

• Why enterprise AI keeps stalling — and how data streaming could unlock it: Megan Carnegie writes: “Enterprise AI is running into a problem. But it has more to do with data infrastructure than model quality. The snag is that most businesses still keep data fragmented across databases, SaaS tools, warehouses, and internal platforms, with security controls layered on top to protect each surface area separately.”

Read Megan’s piece on TheNewStack.

• Kafka Share Groups and Parellelizing Consumption - Part 2: Producer Batches and share.acquire.mode: Jack Vanlighty continues his series of benchmark tests for Apache Kafka Share Groups. The objective: Use synthetic tests to measure the overhead of share groups compared to consumer groups in identical conditions.

Read this latest edition and follow the series on Jack’s blog.

• Improved Column Reader API, First Cut of Geospatial Support: Hardwood 1.0.0.CR1 Is Available: This first candidate release of Hardwood 1.0 brings a substantially improved API for columnar access to Apache Parquet files, initial support for Parquet’s GEOMETRY/GEOGRAPHY column types, and many other improvements to the core library as well as the Hardwood CLI.

Learn more here.

Links From Around the Web:

• IBM and Red Hat Commit $5 Billion to Redefine the Future of Open Source in the AI Era. Project Lightwell will establish a trusted enterprise clearinghouse combined with a global force of engineers to identify and fix vulnerabilities at scale. The clearinghouse will serve as a security coordination layer, using advanced AI capabilities to validate and test fixes across an unprecedented volume of open source code. Learn more about Project Lightwell and read the press release.
• Announcing Apache Iceberg 1.11.0: Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. Visit the release notes to learn more.

In-Person Meetups

• June 9th: Vienna
• June 11th: London
• June 12th: Munich
• June 16th: Paris
• June 17th: Jakarta
• June 18th: London
• June 20th: Coimbatore
• June 20th: Bangalore
• June 24th: Melbourne
• June 30th: Seoul

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, send us an email at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the signup form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.