Announcing Confluent Platform 8.0

Confluent has announced the launch of Confluent Platform 8.0 - the largest Confluent Platform release yet!

This release builds upon Apache Kafka 4.0, further reinforcing Confluent Platform’s core features that make hybrid and on-premises data streaming secure, scalable, and resilient, while also introducing innovative functionalities.

This launch focuses on removing operational complexity, strengthening data protection, and scaling Kafka more efficiently, with updates that modernize how you monitor clusters, secure sensitive data, and eliminate the overhead of Apache ZooKeeper™️.

With this release, Confluent Platform FlinkSQL is in OpenPreview now, and customers can manage their Flink clusters using the next-gen Confluent Control Center (C3).

New key capabilities:

• Easily monitor and manage your Kafka and Apache Flink® environments, using the next-gen Confluent Control Center (C3).
• Simplify your Kafka architecture and unlock massive scalability by replacing ZooKeeper with KRaft.
• Protect your most sensitive data by encrypting individual fields within messages, using Client-Side field level encryption—now Generally Available (GA).

Registration is LIVE for Current New Orleans 2025!

Registration is open for the biggest event in data streaming: #Current25 New Orleans!

If you're building real-time applications or modernizing your data architecture, this is the event to be at.

✅ 60+ technical sessions

✅ Insights from Kafka, Flink & Iceberg experts

✅ Networking with the global data streaming community

Join us October 29–30 to explore the future of data streaming. Register by August 15 to save $500 off the standard ticket price -> Current New Orleans

Know Your Developer (KYD) – Robert Metzger

In this edition’s KYD section, we chat with Robert Metzger, Staff Software Developer II, at Confluent.

1. Hi Robert! Welcome to the “Know Your Developer” section of the Confluent Newsletter. Please introduce yourself.

Hi developers,

My name is Robert, and I'm living in southwestern Germany, at a place with fast enough internet to work fully remotely.

My career in the data processing space kicked off during a 3-month internship at the IBM Almaden Research lab in Silicon Valley during my bachelor's. At the time, "Big Data" was the big hype. Everybody wanted to use Hadoop and MapReduce, and lots of projects were started for building MapReduce jobs more efficiently.

At the research lab, we were doing the same: benchmarking Hadoop/Hive and building our own processing layer (jaql). After that internship and wrapping up my bachelor's, I went to study at TU Berlin, where I got to know the original creators of Apache Flink, back then called Stratosphere. In 2014, we contributed the codebase to the ASF and started pushing on growing Flink's adoption. The rest is history.

2. Tell us a little bit about your background and your journey with Confluent, so far.

My first job was at a company called Data Artisans. I was one of the co-founders of the company, along with some other former students and PhDs from TU Berlin. We started the company in 2014. It was formed to commercialize Apache Flink, provide support, proprietary software, and cloud offerings for it.

In 2018, Flink had probably reached product-market fit: It was adopted by large companies like Netflix, Uber, Alibaba, as well as some larger banks, and was growing quickly. Data Artisans saw its first customers.

In early 2019, Alibaba Cloud acquired data Artisans. After 3 years at Alibaba (still working on Flink), I joined a US-based startup called Decodable to build a Flink cloud product.

After 2 more years, I decided to join Confluent, together with one of my former co-founders from data Artisans times, to launch the on-prem offering for Apache Flink from Confluent, called "Confluent Platform for Flink."
I really enjoy working at Confluent. It's an interesting challenge to build a new product from scratch in a larger organization (at least with my startup background) and turn Confluent into a multi-product company.

3. You have been an Apache Flink committer for a long time! Please share your experience of getting introduced to Flink and how your data streaming journey started?

As mentioned in the beginning of the interview, I got introduced to Flink (back then still called Stratosphere) in like 2012, when I was a student working on experimenting with a novel join algorithm called G-Join (generalized join) for a batch processing framework called Stratosphere. Two years later, I found myself helping to donate the Stratosphere codebase as Flink to the Apache Software Foundation: We wrote an incubator proposal, went through the incubation process, and graduated as a top-level project. As part of the incubation process, I was a PPMC (podling project management committee) member. Podling is the name of projects in the incubator. With the graduation, I became a regular PMC member of Flink.

At that time, Flink was only exposing APIs for batch processing, not streaming. The processing engine was designed to follow best practices from database engines, such as pipelined execution, pluggability of all major components, or using optimizers. Those best practices came in super handy when some new contributors showed up, proposing a stream processing API for Flink. We saw adding stream processing as a good opportunity to expand the scope of Flink, but we initially kept our focus on batch processing—until we realized that stream processing was the feature that distinguished Flink from other data processing frameworks and that the majority of our users were actually looking for these capabilities. Flink offered much more than the leading system at the time: Apache Storm. Flink has a better consistency and recovery model, event-time management, large state support, and many more, bringing stream processing to a completely new level.

We quickly added a lot of important features like the RocksDB state backend, more connectors, savepoints, etc.

It was a super exciting time, because before the pivot to stream processing, we actually struggled quite a bit to compete in the batch processing space against Apache Spark—with streaming, we got much more traction.

4. Apache Flink has grown tremendously in the last 5-6 years, mostly when cloud platforms surged in demand. How do you see other competing stream processing frameworks evolving w.r.t Flink and why Flink holds an unique edge?

I've seen a number of competitors come and go over the last 10 years. Often new systems have the benefit of heightened expectations of the unknown. This means people tend to overestimate the capabilities of the shiny new thing—unknown gaps will be assumed to be solved perfectly fine.

When people hear it's written in Rust, or by somebody who's done a popular system before, or some marketing claiming 10x performance improvement for some specific workload/scenario, people imagine the system will solve all their problems at no cost. But after some time, reality kicks in, and the new system has zero ecosystem integrations, or only works on the happy path, or is not flexible enough for your use case.

Flink holds a unique edge by its flexibility and pluggability. You can hook into many aspects of the system and can configure a lot. On top of that comes the extensive use in a lot of different environments and workloads. This makes the system super robust.

But of course, the competition never sleeps, and we need to pay attention to not miss important new developments. That's why I'm excited to see efforts in Flink around building a new state backend that can leverage blob stores like S3 to maintain infinite state, or a new repo for flink-agents, exploring Flink for the use of agentic AI.

5. What would be your recommendation for a budding developer who has just started on the journey of becoming a data streaming engineer?

Get familiar with Flink's DataStream API to know what it offers, understand the Flink fundamentals, and get really good with Flink SQL. Check out the process table functions (PTF) effort in Flink SQL.

Why? DataStream API is a very popular API in Flink. It's fairly easy to learn and is a nice way for learning the most important concepts of Flink, such as state management and event-time handling, hands-on. However, for most use cases, DataStream API is actually not needed, because you'll be implementing faster with Flink SQL, and it will give you the same or even better performance than DataStream API.

If you are hitting a limit with Flink SQL, PTFs will come in handy to unblock yourself. Or, as a last resort, DataStream API.

Another important project to look into is Apache Iceberg as a bridge between the real-time and batch world.

6. Share with our readers how do you see the future of Apache Flink evolving?

Flink's APIs are where I anticipate most of the innovation: Flink SQL is great, as it gives developers an extremely productive API (little code, big impact). The resulting Flink jobs are very efficient from the operators, data types, etc. For simple use cases, there's really no point in using the DataStream API. It will take a lot of time to come up with the same result as Flink SQL. However, Flink SQL is not done yet—process table functions are an important enabling building block to allow adding a lot more functions to Flink SQL: for all kinds of windowing, joining, aggregation operators, or interactions with other systems. I assume that this will allow Flink SQL to cover more use cases. I also believe that there are some open questions around coming up with efficient Flink jobs from Flink SQL. Sometimes, our developers have to change their SQL queries to improve the execution efficiency—this shouldn't happen in SQL. The optimizer should rewrite the query accordingly. I suspect that some kind of online optimizer which observes the runtime behavior of Flink, and on the next job restart (either due to some failure or an autoscaling event), deploys an optimized plan of the job.

Another area in the API realm is PyFlink. We have a good foundation, but I don't think we've addressed the developer market potential there. Python is everywhere, especially in AI and the wider data processing community. Giving them access to powerful stream processing in their lingua franca.

For both PyFlink and programmatic SQL, I see Flink's Table API as a great tool: It combines a high degree of customization through custom code with the efficiency of Flink SQL. Like Flink SQL itself, Table API needs some iterations to be on par with the DataStream API, but there are no big roadblocks.

Besides the API, the engine itself will also evolve. To name a few examples: there's an effort to support multiway joins, build an "infinite state" state backend, or build autotuning capabilities. Multiway joins greatly improve the performance when streaming data from multiple sources into a common stream, for example when combining CDC streams from many tables into a single result stream.

Infinite state offers a different set of tradeoffs for the state backend: It makes error recovery and rescaling downtimes go away, at the potential expense of (initially) slower processing.

Autotuning is happening in the Flink Kubernetes operator (really nice component of Flink ;)) alongside the autoscaling efforts—but instead of changing the individual Flink operator parallelisms or number of TaskManagers (workers), autotuning is changing Flink's configuration (for example for memory allocations) to incrementally learn from observed runtime metrics.

To wrap up, Flink's future is bright, and I'm super excited to see all these developments end up 🙂

Thanks a lot for this interview!

Data Streaming Resources

• How do event-driven architecture, stream processing, orchestration, and the new durable execution category relate to one another? Read Coordinated Progress, written by Jack Vanlightly, Principal Technologist - Confluent, a 4-part series describing how he sees it and his mental framework.
  • Coordinated Progress – Part 1 – Seeing the System: The Graph
  • Coordinated Progress – Part 2 – Making Progress Reliable
  • Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity
  • Coordinated Progress – Part 4 – A Loose Decision Framework
• Lenses.io introduced the Lenses SRE Agent which provides intelligent triage to help teams quickly focus their efforts on Kafka troubleshooting. It helps to:
  • Analyze message patterns
  • Provide historical context
  • Identify schema issues
  • Read the blog from lenses.io

Links From Around the Web:

• Google Cloud donates Agent2Agent (A2A) protocol to Linux Foundation. The project will be hosted by the Linux Foundation and will be seeded with Google’s transfer of the groundbreaking Agent2Agent (A2A) protocol specification, accompanying SDKs, and developer tooling. The A2A protocol, an open standard for communication and collaboration between distinct AI agents, aims to break down the silos that currently limit the potential of artificial intelligence. More than 100 companies now support the protocol, with AWS and Cisco as its newest validators.
• Watch a short video, where the CEO of Neon DB, Nikita Shamgunov speaks about improving the Developer Experience of a database and the innovations which made it possible!

In-Person Meetups:

• July 26th - Bangalore
• July 28th - Cleveland w/Elastic
• Aug 19th - VIRTUAL Austin w/Python Group
• Aug 28th - Belgium w/Cymo

Stay up to date with all Confluent-run meetup events - by copying the following link into your personal calendar platform:

https://airtable.com/app8KVpxxlmhTbfcL/shrNiipDJkCa2GBW7/iCal

(Instructions for GCal, iCal, Outlook, etc.) 

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.