Design events, solve the Dual Write problem… all that and more in this week’s roundup!

The Dual Write problem has long been known to make developers quake in their boots. The transactional outbox pattern is your way out! This week we’re featuring Wade Waldron’s video on this pattern, as well as a blog post about using Confluent for K8s.

We’ve also got a LinkedIn article on Sliding Windows in Apache Kafka® and OVER aggregations functionality in Apache Flink®. This article points out a logical similarity between Kafka Streams and Flink, and it got us thinking, are there more logical similarities out there? If you know of one, send it on over to devx_newsletter@confluent.io!

Apache Kafka® Resources:

• How to Use Confluent for Kubernetes to Manage Resources Outside of Kubernetes - Tomás Dias Almeida discusses “how to use Confluent for Kubernetes (CfK) to enable GitOps with a CI/CD pipeline and delegate resource creation to groups of people without distributing admin permission passwords to other people in the organization.” There’s a demo you can use to follow along!
• Designing Events & Event Streams - You put WHAT in your events? It’s not just the partitioning and cluster configuration that counts when you’re event streaming… Learn how to design events and event streams in a way to minimize your pain with Adam Bellemare’s course.
• The Transactional Outbox Pattern - If you’ve been slamming into the dreaded Dual Write problem, here’s one solution! Wade Waldron walks you through it.
• A single message key to field transform for Kafka Connect/Debezium by Elad Leev! In a way it’s the reverse of the ValueToKey SMT that comes with Kafka Connect, useful for when the fields in the key are not included in the value.

Apache Flink® Resources:

• Mastering Stream Processing: Sliding Windows and OVER Aggregations - See how Sliding Windows in Kafka and OVER aggregations in Flink are both similar and different in this LinkedIn post by Bill Bejeck
• An opinionated approach to forking Flink: the second installment of the series by Yaroslav Tkachenko features class shadowing.

A Droplet From Stack Overflow:

This week, see David Anderson’s answer to the question: how can I understand the codebase better? There’s a list of helpful links, including a lineup of talks on Flink!

• Nico Kruber’s talk on the network stack
• Stefan Richter’s talks on state and checkpointing
• Piotr Nowojski’s talk on two phase commit sinks

Got your own favorite Stack Overflow answer related to Flink, Kafka, or Streaming Infrastructure? Send it in to devx_newsletter@confluent.io!

Terminal Tip of the Week:

Did you know that you can view the AST (Abstract Syntax Tree) of the query planner and the full detailed execution plan for your SQL statements in PyFlink? This is how:

On your table environment instance, call the function explain_sql like so:

table_env.explain_sql("SELECT * FROM table")

Following that, you will receive the following output:

== Abstract Syntax Tree ==
LogicalProject(window_start=[$0], window_end=[$1], price=[$2])
+- LogicalTableScan(table=[[default_catalog, default_database, table]])

== Optimized Physical Plan ==
TableSourceScan(table=[[default_catalog, default_database, table]], fields=[window_start, window_end, price])

== Optimized Execution Plan ==
TableSourceScan(table=[[default_catalog, default_database, table]], fields=[window_start, window_end, price])

Perhaps you don’t remember this, but last week we asked you to spot the error in the following CC Flink SQL query:

CREATE TABLE Treat (
  `rice_krispy` STRING,
  `canoli` STRING,
  `gulab_jamun` STRING
) WITH (
  'connector' = 'kafka'
  ...
);

The culprit here is that ‘treat’ is a reserved keyword, and it’s best avoided, eliminating confusion.

Links From Around the Web:

• A codepen showing you how to use depth-scrolling
• How to debug browser redirects
• Learn about generative art design

Upcoming Events:

In-person:

• In-person meetup in Gurgaon (Mar 16): Apache Kafka® Meetup in Gurgaon - March 2024

Hybrid:

• Current 2024 (Sep 17-18): The call for papers for Current 2024 is now closed. Thank you for your submissions! We will keep you updated on speaker and agenda announcements as they become available.
• Kafka Summit London (March 19-20): Last call for London! Registration for KSL is still open! Planning on attending? Now is a good time to check out the agenda and speakers for the event and planning which talks are your must-sees!
• Kafka Summit Bangalore (May 2): Registration for KSB is now open! While we wait for the first-ever Kafka Summit in Bangalore, you can check out the agenda and speakers for the event.

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for us to share with the community, email us at devx_newsletter@confluent.io!