Using Apache Flink® to Identify Cheat Codes

Newsletter from the Desk of Confluent Developer,

This week's resources include a deep dive into Apache Kafka® producer internals and more!

In this week’s newsletter, we have data streaming goodies galore! First, a technical treatment of Kafka producer internals written by Danica Fine, then z-shell autocomplete for the Kafka CLI from Neil Buesing! Dan Weston shows us a way to deal with paused streams in Flink, and Dale Lane gives us a fun example of MATCH_RECOGNIZE. Lastly, an example project lovingly created by Barry Evans rounds out our resource list. Don’t miss the meetups section-- there are 15-- count 'em, 15! meetups this month.

Data Streaming Resources:

• Dive in to Kafka internals with Danica Fine’s latest post on Kafka producers and how they work

• Confluent Community Catalyst Neil Buesing brings us z-shell autocomplete for the Apache Kafka CLI!

• Dan Weston shows us a solution for handling paused streams in Flink in the latest streamable video

• Dale Lane explains using Flink to recognize video game cheat codes. A fun example of MATCH_RECOGNIZE.

• Barry Evan’s example project teaches you about SMTs, Spring Boot, and more!

• The latest installment in Bill Bejeck’s stream processing series: on viewing and analyzing results

A Droplet From Stack Overflow:

Builds can be tricky… if you’ve ever tried to build flink-avro and you’ve noticed it downloading the same things you’ve built locally… Matthias J Sax has an answer for you!

Got your own favorite Stack Overflow answer related to Flink or Kafka? Send it in to devx_newsletter@confluent.io !

Terminal Tip of the Week:

To add the auto-complete functions from Neil Buesing’s kzsh-autocomplete tool, follow these instructions.

Add a Directory for Custom Z-Shell Functions

While these auto-complete functions can be added to an existing function directory, it is best to separate them to avoid losing them on an OS or shell upgrade.

mkdir -p ~/.zsh/functions/

Add This Directory to Your fpath

While this could easily be done with adding the following to your .zshrc file, it is best to set up changes to .zshrc file that don't add the directory multiple times when re-sourced.

fpath=(~/.zsh/functions $fpath)

Instead, find current fpath by doing echo $fpath and then creating one as follows:

fpath=(~/.zsh/functions /usr/local/share/zsh/site-functions /usr/share/zsh/site-functions /usr/share/zsh/5.9/functions)

Enable Z-Shell autocomplete

If you are using a zsh framework, read up on their documentation to determine their way of enabling autocomplete. Typically, this is done by adding the following to your ~/.zshrc file.

autoload -Uz compinit
compinit

The rest of the instructions can be found in the README.

Links From Around the Web:

• A browser extension that visualizes how much each person is talking in Google Meet

• Reuter’s scrollytale on the collapse of insects

• Julia Evans’s notes on git error messages

Upcoming Events:

In-person

• Paris, France, (Oct 3): Lucia Cerchie takes us on a tour of Kafka and Flink for data visualization, and David Pilato introduces ES/QL.

• Kruibeke, Belgium (Oct 9): Benjamin Barrett demystifies RocksDB, which backs statestores in Kafka Streams and Danica Fine dives deep into Kafka consumer internals.

• Cologne, Germany (Oct 29):Learn about Service Level Objectives (SLOs) in a distributed system.

• Birmingham, UK (Oct 24): Come to this meetup to hear about vector and hybrid search.

• Berlin, Germany, (Oct 22): Fast aggregation and real-time insights will be the topic of the evening.

• Tokyo, Japan, (Oct 18): Come to this Kafka meetup in Tokyo!

• Belfast, Ireland, (Oct 17): Learn about using Mockstream to create mock data streams for demos.

• Taipei, Taiwan (Oct 15): Building AI apps with Kafka and Flink!

• Munich, Germany, (Oct 10): See how Kafka + Druid work together for creating an analytics pipeline.

• Brussels, Belgium (Oct 9): Dive into internals: clients and how they work taught by Danica Fine, and what RocksDB is and how it supports Kafka Streams.

• New York City, USA (Oct 8): Kafka, Flink, and Pinot: Viktor Gamov presents!

• London, UK (Oct 8 ): Infrastructure as code for event streaming products, and adopting Kafka in a hybrid world.

• Utrecht, Netherlands (Oct 8 ): Migrating from KRaft to zookeeper and how Alliander effectively uses Kafka.

• Brussels, Belgium (Oct 8 ): 2 talks on 3 technologies each: Kafka, Flink + Druid, then RUM, Otel, + Elastic

• Denver, CO, USA (Oct 8 ): Deep dive into Kafka Streams and FlinkSQL with Bill Bejeck and Matthias J. Sax

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.