Hey there, we hope the month of May treated you well. As we usher in the month of June, we take a look at some interesting resources, both in technical detailing and sweeping architectural overviews. There’s the inaugural episode of The Duchess and the Doctor show (what’s that? Read on…) and a blog post outlining how to use the OpenSearch sink connector. Enjoy!

Data Streaming Resources:

• Watch The Duchess and the Doctor show: A quarterly release in which Anna McDonald and Matthias Sax answer your burning questions about event-driven architecture.

• Learn how to use OpenSearch connector for Kafka by following along the steps in this blog post.

• Best practice for End to End building data streaming pipelines on Confluent using Terraform: read this blog post to avoid the pitfalls

• Learn the ins and outs of how Oso scaled from a single instance to a globally available service in 12 regions using Kafka

• Ever wondered what deal is with the auto.offset.reset on the Kafka consumer? Read Danica Fine’s new blog post

• Tributary Bank: A new case study from our event-driven microservices course

A Droplet From Stack Overflow:

Curious on how Kafka python producer capabilities such as flush() and a poll()? Are you wondering what the difference is between them? Learn more here.

Got your own favorite Stack Overflow answer related to Flink or Kafka? Send it in to devx_newsletter@confluent.io!

Terminal Tip of the Week:

Today’s example focuses on the kafka-metadata-quorum CLI. It’s an invaluable tool to debug a kafka cluster in KRaft mode. Pass the describe parameter to get the full description of the metadata quorum current state.

Here is a 3 node KRaft kafka cluster example:

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9097 describe --status

The output of the command displays the following status:

ClusterId:              1h3bN31wQAqDRNu7du_oFg
LeaderId:               1
LeaderEpoch:            2
HighWatermark:          212
MaxFollowerLag:         1
MaxFollowerLagTimeMs:   492
CurrentVoters:          [1,2,3]
CurrentObservers:       []

This information provides a snapshot of the cluster metadata status to help you understand if the cluster is in a consistent state or not. For example, LeaderEpoch can help you figure out if the in-sync replicas need log reconciliation after a new leader election.

Here’s the usage details of the kafka-metadata-quorum CLI :

USAGE: kafka-metadata-quorum.sh [-h] --bootstrap-server BOOTSTRAP_SERVER [--command-config COMMAND_CONFIG] {describe}
                            ...

This tool describes kraft metadata quorum status.

positional arguments:
  {describe}
    describe             Describe the metadata quorum info

optional arguments:
  -h, --help             show this help message and exit
  --bootstrap-server BOOTSTRAP_SERVER
                        A comma-separated list of host:port pairs  to  use  for establishing the connection to the
                        Kafka cluster.
  --command-config COMMAND_CONFIG
                        Property file containing configs to be passed to Admin Client.

Links From Around the Web:

• If programming languages were games… what games would they be? A webcomic
• An interactive blog about how Bezier Curves work
• The world’s largest flipbook animation. See the results.
• You’ve heard about ChatGPT… how about ChatGPTea?

Upcoming Events:

Hybrid

• Current 2024 (Sep 17-18): The agenda for Current 2024 has been released. Check it out! Register here to attend.

In-person

• Tracing Kafka with OpenTelemetry & Streams Don’t Fail Me Now - Robustness Features in Kafka Streams (June 4): Harry Kimpel and Lucas Brutschy will, respectively, tell us about tracing Kafka and making KStreams robust. Nürnberg, BY.
• Apache Kafka® 101 (June 11): Get an introduction and overview of Kafka from Brice Leporini in Doha
• Reverting and Resolving Issues in Kafka and Iceberg Environments (June 12th): Angus Neilson and Chris Finlayson give us the details on Kafka and Iceberg. Located in Edinburgh.
• Data enrichment patterns with Apache Flink (June 5): Subham Rakshit from AWS will speak on enriching streaming data on June 5 at Community Over Code in Bratislava, Slovakia.

Online

• JS Builders Meetup #01 with Lucia Cerchie: Lucia Cerchie, Developer Advocate, will teach about about Kafka is and why we should care. She will go over the fundamentals: the how, why, and when to use Kafka. Then we'll get into some code and learn how to send and receive messages to Kafka topics using a JavaScript client.

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.