Introducing Apache Kafka® 4.1.0

Apache Kafka® 4.1.0 contains 18 Kafka Improvement Proposals (KIPs) from both Confluent and the broader Apache Kafka community. The release hardens 4.0 and brings new capabilities for Kafka operations, Streams, Connect, Queues, KIP-848, OAuth security, and more. While there's a lot in Apache Kafka 4.1.0, here's a summary of some of the key highlights. Please see the Apache announcement, Confluent's blog, or release video to learn about all of the great innovation included in Apache Kafka 4.1.0.

Highlights:

KIP-932 Preview: Queues for Kafka – Queuing capabilities native in Kafka through the new share groups and share consumer that decouple consumer scaling from topic partitions, enabling cooperative message consumption with individual acknowledgments and retries.

KIP-1071 Early Access: Streams Rebalance Protocol – Built on the innovation from KIP-848, KIP-1071 is a broker-driven rebalancing system specifically designed for Kafka Streams applications, providing streams-optimized coordination separate from the traditional consumer group protocol. Early Access (EA) guidance will be provided in Apache docs (see this PR for details).

Hardening & Follow-on Work
Enhanced reliability through follow-on improvements to KIP-853 (KRaft membership controls), KIP-890 (Transactions Defense), and KIP-848 (Consumer Group Protocol).

Flink SQL Query Profiler in Confluent Cloud for Apache Flink® launched!

The first version of Query Profiler, a powerful new observability tool for Confluent Cloud for Apache Flink, has been launched!

Developing and managing complex streaming applications can be challenging, and pinpointing performance issues so far hasn't been possible for our customers due to the lack of any visibility into their seemingly black-boxed statements.

The Query Profiler tool solves this by providing a visual representation of their CC Flink statement's job graph with detailed, real-time metrics to help them identify and resolve performance impediments.

This first release is focused on providing a foundational, visual-first experience for statement analysis:

• Visual Job Graph: Get a real-time visualization of your Flink statement's execution plan. See how your data flows between different tasks and operators and understand the structure of your application at a glance.
• Detailed Task-Level Metrics: Click on any task in the graph to open a side panel with a detailed breakdown of its performance. Metrics include throughput, state size, parallelism, busyness, backpressure, and more.
• At-a-Glance Health Indicators: Task boxes in the job graph are color-coded based on their 'busyness' metric, allowing you to instantly spot which parts of your statement are computationally heavy.

The Query Profiler is available for any user who can view and manage Flink statements in Confluent Cloud. Simply navigate to a running statement to find the profiler and begin exploring the Job Graph and its metrics. Docs can be found at Flink SQL Query Profiler in Confluent Cloud for Apache Flink | Confluent Documentation

Registration is LIVE for Current New Orleans 2025!

Registration is open for the biggest event in data streaming: #Current25 New Orleans!

If you're building real-time applications or modernizing your data architecture, this is the event to be at.

✅ 60+ technical sessions

✅ Insights from Kafka, Flink & Iceberg experts

✅ Networking with the global data streaming community

Join us October 29–30 to explore the future of data streaming. Register by August 15 to save $500 off the standard ticket price -> Current New Orleans

Data Streaming Resources

• WarpStream makes the Bring Your Own Cloud (BYOC) model safer and more secure than any alternative. This was a deliberate design choice that was made possible by WarpStream's Zero Disk Architecture. With a zero-trust BYOC model, WarpStream customers truly get the best of both worlds: an (almost) fully managed user experience, but with all of the cost and security benefits of running on their own infrastructure. Read more here.
• Check out this new Oracle blog, Oracle Autonomous Database & Confluent Tableflow: Real-Time Kafka Analytics Without ETL, written by Oracle's Iceberg PM team and co-authored by Matthew O'Keefe, Principal Technologist at Confluent, and Scott Sappenfield, Partner Solutions Engineer at Confluent, highlighting how customers can use Tableflow to automatically materialize Kafka topics as Iceberg tables in Amazon S3. Oracle Autonomous Database (ADB) can then treat those tables as external tables, so data engineers can run rich Oracle SQL (or even Select AI) across real-time events without copying any data!

Links From Around the Web:

• While protocols like Anthropic's Model Context Protocol (MCP) and Google's Agent-to-Agent (A2A) have made significant progress in enabling seamless communication between AI agents, there is a need to address the larger architectural challenges required to create a true "Internet of AI Agents." At MIT, a team has been pioneering this field, and their latest initiative—the NANDA (Networked Agents and Decentralized AI) architecture—is designed to tackle these challenges head-on. Read here, as Prof. Ramesh Raskar from MIT explains what "NANDA" is. Here’s the GitHub repository for Project Nanda!
• The MCP Gateway is Docker's open-source, enterprise-ready solution for orchestrating and managing Model Context Protocol (MCP) sservers securely across development and production environments. It is designed to help organizations connect MCP servers from the Docker MCP Catalog to MCP clients without compromising security, visibility, or control. Read the docs.

In-Person Meetups:

• 25 September 2025 - New York City
• 25 September 2025 - Warsaw
• 25 September 2025 - Kuala Lumpur
• 27 September 2025 - Bangalore
• 30 September 2025 - Dallas
• 30 September 2025 - Paris
• 1 October 2025 - Denver
• 1 October 2025 - Bay Area
• 1 October 2025 - Lisbon
• 1 October 2025 - Dublin
• 2 October 2025 - Minneapolis
• 6 October 2025 - Bay Area [Menlo Park]
• 7 October 2025 - Dublin
• 7 October 2025 - Brussels
• 8 October 2025 - Jakarta
• 9 October 2025 - Melbourne
• 14 October 2025 - Berlin
• 15 October 2025 - Denver
• 15 October 2025 - Amsterdam

Stay up to date with all Confluent-run meetup events - by copying the following link into your personal calendar platform:

https://airtable.com/app8KVpxxlmhTbfcL/shrNiipDJkCa2GBW7/iCal

(Instructions for GCal, iCal, Outlook, etc.) 

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.