Flink SQL Workspaces - New Features

A major set of features was shipped for Flink SQL Workspaces on Confluent Cloud. These features greatly enhance the user experience when writing sophisticated Flink SQL streaming jobs. Some highlights of the features released are:

1. Pre-submission error highlighting: The SQL editor now highlights any errors in your SQL before you even submit a statement. The precise cause of an error is also underlined with red squiggly lines, to help you quickly determine what needs to be fixed. This feature eliminates the cognitive overhead of visually parsing SQL to figure out what's wrong with an incorrect statement, while also giving the UI a much lighter feel - by reducing statement submission round trips.

2. Tabbed workspaces: Previously, only a single SQL Workspace could be open at any given time. Each Workspace is now its own in-browser tab that maintains state across tab visits, enabling users to run many more statements with less effort. And with a greatly improved SQL Workspaces landing page, users have multiple jumping off points preconfigured for common purposes (query a topic with auto- generated SQL, load an example Workspace, etc.).

3. Time-series visualization: The output of certain SQL statements may now be seamlessly rendered as time-series charts. Whenever a statement's output has at least one time column, and at least one numeric column, we will automatically chart it in time-series when the user toggles to chart mode. Charts may be further customized by user interactions: users can choose a different x-axis column, add multiple series, change the chart's time granularity, filter the overall time range, etc. One of the really great aspects of how this works is that chart mode is fully integrated with table mode: filters applied to the table also apply to the corresponding points in the chart, providing a delightfully fluid analytical experience.

Current 2025 Bengaluru - Registrations Open!

Current 2025 Bengaluru, the first edition of the world’s most popular data streaming event, will take place on the March 19th, 2025, at Sheraton Grand Bengaluru Whitefield.

Registrations are now open for everyone! Register now, and witness the biggest data streaming event of the year in India!

Know Your Developer [KYD] - Matthias J. Sax

In this edition’s KYD section, we chat with Matthias J. Sax, Staff Software Engineer II at Confluent.

Matthias is an Apache Kafka committer and PMC member. He is also a committer at Apache Flink and Apache Storm. Matthias' name is synonymous with the Kafka Streams library. Let’s hear about his journey so far.

1. Hi Matthias! Welcome to the “Know Your Developer” section of the Confluent Newsletter. Would you like to introduce yourself?

Hi, I was born and raised in Bavaria, which is the best state of Germany. In my teenage years, I started programming with Basic and Pascal, semi-successfully taught myself C, and failed at C++ (which I only picked up later). Currently, I am based in Reno, which is not ideal for the surfing that I love, but great for Texas Hold’Em. Of course, I am a big football fan (super FCB!), and follow American (fake) football nowadays, too. Go Bills!

2. Tell us a little bit about your background and your journey with Confluent, so far.

I studied computer science at the Humboldt University of Berlin, and joined Confluent as a new grad, if you wish. Confluent was originally not super appealing to me, as I wanted to work on data processing, but Kafka was only a storage system at this time. During the interview I learned about Kafka Streams which was released shortly after. I guess, the rest is history. Naturally, I also worked on ksqlDB.

Besides coding, I have always enjoyed teaching, and so I am very active in the open-source community. It’s great that Confluent supports these efforts, and we clearly benefit from it, because understanding your users helps to build better software.

3. Your name is synonymous with the open-source Kafka Streams framework! Tell us how you gained experience working on distributed systems and streaming data?

It all started at the university, where I focused on databases and distributed systems, and interned with IBM (DB2) and SAP. In my post-graduate years, I shifted to stream processing, as part of the Stratosphere research project, which later became Apache Flink; and I also worked with Apache Storm during this time. My academic background, lessons learned from real systems like Flink and Storm, and the pragmatic startup culture at Confluent, were an ideal mix to shape Kafka Streams into what it is today: “The best streaming library in the world.” -- A.McD.

4. If you were to start again as a developer/engineer, would you do something differently? Please share your opinions with our readers.

Overall, I think I would not change much, but it highly depends on the type of person you are. For me, spending more than 10 years at the university was perfect, as it gave me the time to really dive into data processing and distributed systems fundamentals. But such a setting is not ideal for more hands-on people, who prefer to explore and learn on their own. Overall, I would recommend that everyone spend time on the basics, otherwise you build your career on sand. Books like “Designing Data-Intensive Applications” are a great resource to strengthen this muscle.

5. There is a conscious effort to converge the streaming and batch processing paradigms. What is your take on this, and how do you see this space evolving?

I have a less “optimistic” opinion: I believe there will always be use cases for batch processing, including querying datasets. Data warehouses/lakes won’t go away, and while data ingestion will become streaming, querying is its own domain that stream processing cannot replace.

Also, the semantics of both are very different, and I doubt there is actually much to be unified for the general case. I have discussed this a lot with Anna McDonald of Confluent, and the fact that you cannot “pause” in stream processing, makes it very hard to mimic a classic step-by-step batch workflow: “after (A) is done, start (B)” is not a concept in streaming, and often such step-by-step workflows really need this “pause” in between. Anna coined the term “event driven cohorts,” which allows for batch-like semantics in streaming (note, the reversed direction!), but it’s of course more difficult to set up, run, and reason about, compared to a batch workflow.

In the end, streaming solves a harder problem, and thus requires a more sophisticated solution, and not all applications or pipelines need the benefits of stream processing. No value proposition; no adaptation. Of course, there are a huge number of use cases that benefit from low latency, incremental processing, and stream processing will become the dominant solution there.

6. Share with our readers how you see Kafka Streams evolving, i.e., the future of the framework?

Kafka Streams has a bright future ahead, and some newly founded startups even couple their own existence and success to Kafka Streams. At Confluent, we are currently focusing on strengthening the fundamentals, for example, with the new streams consumer group protocol and transactional state stores, or the already improved threading model. At the same time, Kafka Streams provides all of the building blocks to support richer verticals like workflow/durable execution that the aforementioned startup built with Kafka Streams. Furthermore, one of the most requested “features” is to offer Kafka Streams in other languages. This will be a heavy lift, but it’s just a natural consequence of its success in the JVM world, and its nature of it being a library. -- Long live the otter.

Data Streaming Resources:

• Grafana Mimir — the open-source, horizontally scalable, multi-tenant time series database (TSDB), has undertaken architectural changes to Mimir and Grafana Cloud Metrics—the fully managed, highly scalable metrics service powered by Mimir. It is making them more reliable, so they can support future growth and new use cases at an even larger scale. To achieve the core objective of decoupling read and write paths, WarpStream caught the attention of the Grafana team, because it’s a Kafka-compatible data streaming platform built directly on top of object storage. Read their blog to learn how the team wrote 1.5 GiB/s (compressed) through a single WarpStream cluster and consumed it with 4x fan-out, for an aggregate compressed throughput of 7.5 GiB/s with no bottlenecks, demonstrating that WarpStream could meet their scaling needs.
• AI agents are the latest talk of the town! Read this exciting blog on event driven multi-agent systems, written by Sean Falconer and Andrew Sellers of Confluent, where they talk about reasoning agents and how reasoning agents work together in multi-agent systems.
• Check out the most viewed documents on Confluent Cloud and Confluent Platform:
  • Apache Flink - A Complete Introduction
  • Introduction to Python for Kafka
  • Apache Kafka®: Basic Concepts, Architecture, and Examples

Links From Around the Web:

• IBM to Acquire DataStax, Deepening watsonx Capabilities and Addressing Generative AI Data Needs for the Enterprise. Read here.
• Apache Flink 1.20.1 release announcement - The first bug fix release of the Flink 1.20 series has been released ! This release includes 75 bug fixes, vulnerability fixes, and minor improvements for Flink 1.20. For a complete list of all changes see: JIRA. All users are highly recommended to upgrade to Flink 1.20.1.
• DataHub 1.0 is released! This release deepens DataHub’s commitment to open source and also adds exciting features like DataHub Iceberg REST Catalog beta!

Catalyst Insight:

In our brand-new “Catalyst Insight” section, we ask catalysts from the data streaming community to share their experiences.

In this edition, we ask Zabeer Farook to give his insights. Zabeer is a seasoned Technology Architect, with close to 20 years of experience in the IT industry. He has strong skills in Technical Architecture and the Design & Development of Enterprise Software Applications.

How would you describe your role in the data world? Not necessarily as in your title, but what unique perspective and experiences do you bring?

As a seasoned data streaming professional, I've developed expertise in designing complex streaming solutions across diverse use cases, like CDC-based, real-time data synchronization, event-driven architectures, application monitoring, audit tracking, and streaming ETL pipelines. My experience spans technologies such as Kafka Streams, KSQL, and Flink, with a deep commitment to leveraging stream processing to drive real-time data accuracy. I firmly believe that timely, precise data is fundamental to delivering exceptional user experiences—after all, users demand immediate insights, and cannot tolerate inaccurate or outdated information. By implementing robust streaming architectures, we can transform raw data into meaningful, up-to-the-moment intelligence that directly impacts user satisfaction and business value.

Can you tell us the story of an interesting data streaming bug you ran into and solved at one point?

In a complex CDC streaming data pipeline bridging a legacy and modern system, we encountered a critical synchronization bug, where delete and create events occurred in an unexpected sequence. The legacy system generated a delete event, followed by an immediate create event, contrary to the standard operational pattern. To address this, we implemented a staging mechanism that temporarily holds delete events when no corresponding object exists in the target system. An independent retry flow with exponential backoff was designed to manage these staged events, ensuring eventual data consistency. This experience highlighted the importance of building flexible event handling strategies that can accommodate unpredictable legacy system behaviors.

What advice would you offer a burgeoning data streaming engineer?

When designing streaming data pipelines, look beyond just functional requirements. Pay equal attention to data quality, robust governance frameworks, well-defined data contracts, and performance optimization. These aspects are crucial for creating reliable, scalable, and efficient streaming solutions that deliver real business value. Treating data as a first-class citizen ensures that your pipelines not only work technically, but also maintain integrity, compliance, and operational excellence.

Want to learn more about our Confluent Community Catalyst Program? Visit the page here to get all the details!

Upcoming Events:

In-Person Meetups:

• March 6th - Boston
• March 6th - Phoenix
• March 8th - Hyderabad
• March 11th - Lisbon
• March 12th - Seville
• March 12th - Toulouse
• March 12th - Washington D.C
• March 13th - Brisbane
• March 13th - Bangkok

Stay up to date with all Confluent-run meetup events - by copying the following link into your personal calendar platform:

https://airtable.com/app8KVpxxlmhTbfcL/shrNiipDJkCa2GBW7/iCal

(Instructions for GCal, iCal, Outlook, etc.) 

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.