Apache Flink 1.20 has arrived!

Among the improvements are two featured FLIPs that will simplify your interactions with Flink tables:

• FLIP-376 introduces a way to specify a bucketing strategy for a Flink SQL table
• FLIP-435 introduces a Materialized Table abstraction, which lets the engine automatically create and maintain a query result with the requested freshness

Expected to be the last minor release before Flink 2.0, version 1.20 also includes several deprecations, as the community does its final preparations for 2.0:

• The legacy SinkFunction API has been deprecated
• Roughly 30 other configuration options that are no longer useful have been deprecated, and some others have been reorganized, most notably the options concerning checkpoints, savepoints, state, and state recovery.

Get all of the details in the release announcement.

Data Streaming Resources:

• Idle timeouts provide a unique way of dealing with watermarks when data emission velocity is unpredictable. Check out Dan Weston's video on how to set idle timeouts when joining data in Apache Flink®
• Kafka Tiered Storage (KIP-405) was conceived to avoid tight coupling of storage and processing in a broker. Learn about the Kafka Tiered Storage architecture at Uber, here
• How does Notion handle 200 billion data entities? Read the intriguing blog, here
• Complete Course Alert: Designing Event-Driven Microservices:
  • Learn the fundamentals of microservices and their role in building event-driven architectures with this three-hour, intermediate level course from Wade Waldron, Staff Software Practice Lead at Confluent
  • Topics include
    • Data ownership
    • The dual-write problem
    • Event sourcing
    • Schema evolution
    • Asynchronous events
    • A rich set of case studies
    • … and more!
    • Get started on Confluent Developer!

A Droplet From Stack Overflow:

Unknown magic byte!

If you've ever seen this dreaded error when trying to consume from a Kafka topic, the solution, as Robin Moffat suggests in today's answer, lies in using the Schema Registry serializer.

Got your own favorite Stack Overflow answer related to Flink or Kafka? Send it in to devx_newsletter@confluent.io!

Terminal Tip of the Week:

It can be hard to remember lexical precedence when it comes to Flink SQL syntax. Say you had some sort of query involving basic math:

SELECT user from ROW WHERE user\_id \= 5 \* 3 \+ 2

(Why you wouldn't say SELECT user from ROW WHERE user\_id \= 17 is beyond me, this is just for the sake of example.)

Now, in order to make this query clearer for other SQL wranglers to read, you could add parentheses to emphasize that the multiplication operation will be executed first:

SELECT user from ROW WHERE user\_id \= (5 \* 3\) \+ 2

Read more about this in the Confluent docs.

Links From Around the Web:

• Ever wondered why the OR | operator is used in so many programming languages?
• The case of a curious SQL query
• Don't miss a pawesome time at Current, September 17-18, in Austin (courtesy, Dave Troiano from Confluent!)

Upcoming Events: Hybrid:

• Austin: Current 2024 (September 17-18) - The agenda for Current 2024 has been released. Check it out! Register here to attend

In-person:

• Jakarta: August 29 - Hands On Workshop: Automation Deployment of Apache Kafka®
• Sydney: August 29 - IN PERSON! Apache Kafka® Meetup - August 2024
• Auckland: September 4 - IN PERSON! Apache Kafka® Meetup - September 2024
• Copenhagen: September 5 - IN PERSON: Apache Kafka® Meetup - September 2024
• Budapest: September 11 - IN PERSON: Apache Kafka® Meetup - September 2024
• Paris: September 11 - Meetup de rentrée - meshIQ - observabilité Kafka @ Criteo

By the way…

We hope you enjoyed our curated assortment of resources!

If you'd like to provide feedback, suggest ideas for content you'd like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you'd like to view previous editions of the newsletter, visit our archive.

If you're viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.