Get Started Free

Using OpenSearch as an Apache Kafka® Sink Connector and more!

May 30, 2024

Hey there, we hope the month of May treated you well. As we usher in the month of June, we take a look at some interesting resources, both in technical detailing and sweeping architectural overviews. There’s the inaugural episode of The Duchess and the Doctor show (what’s that? Read on…) and a blog post outlining how to use the OpenSearch sink connector. Enjoy!

Data Streaming Resources:

  • Watch The Duchess and the Doctor show: A quarterly release in which Anna McDonald and Matthias Sax answer your burning questions about event-driven architecture.

  • Learn how to use OpenSearch connector for Kafka by following along the steps in this blog post.

  • Best practice for End to End building data streaming pipelines on Confluent using Terraform: read this blog post to avoid the pitfalls

  • Learn the ins and outs of how Oso scaled from a single instance to a globally available service in 12 regions using Kafka

  • Ever wondered what deal is with the auto.offset.reset on the Kafka consumer? Read Danica Fine’s new blog post

  • Tributary Bank: A new case study from our event-driven microservices course

A Droplet From Stack Overflow:

Curious on how Kafka python producer capabilities such as flush() and a poll()? Are you wondering what the difference is between them? Learn more here.

Got your own favorite Stack Overflow answer related to Flink or Kafka? Send it in to!

Terminal Tip of the Week:

Today’s example focuses on the kafka-metadata-quorum CLI. It’s an invaluable tool to debug a kafka cluster in KRaft mode. Pass the describe parameter to get the full description of the metadata quorum current state.

Here is a 3 node KRaft kafka cluster example:

bin/ --bootstrap-server localhost:9097 describe --status

The output of the command displays the following status:

ClusterId:              1h3bN31wQAqDRNu7du_oFg
LeaderId:               1
LeaderEpoch:            2
HighWatermark:          212
MaxFollowerLag:         1
MaxFollowerLagTimeMs:   492
CurrentVoters:          [1,2,3]
CurrentObservers:       []

This information provides a snapshot of the cluster metadata status to help you understand if the cluster is in a consistent state or not. For example, LeaderEpoch can help you figure out if the in-sync replicas need log reconciliation after a new leader election.

Here’s the usage details of the kafka-metadata-quorum CLI :

USAGE: [-h] --bootstrap-server BOOTSTRAP_SERVER [--command-config COMMAND_CONFIG] {describe}

This tool describes kraft metadata quorum status.

positional arguments:
    describe             Describe the metadata quorum info

optional arguments:
  -h, --help             show this help message and exit
  --bootstrap-server BOOTSTRAP_SERVER
                        A comma-separated list of host:port pairs  to  use  for establishing the connection to the
                        Kafka cluster.
  --command-config COMMAND_CONFIG
                        Property file containing configs to be passed to Admin Client.

Links From Around the Web:

  • If programming languages were games… what games would they be? A webcomic
  • An interactive blog about how Bezier Curves work
  • The world’s largest flipbook animation. See the results.
  • You’ve heard about ChatGPT… how about ChatGPTea?

Upcoming Events:




  • JS Builders Meetup #01 with Lucia Cerchie: Lucia Cerchie, Developer Advocate, will teach about about Kafka is and why we should care. She will go over the fundamentals: the how, why, and when to use Kafka. Then we'll get into some code and learn how to send and receive messages to Kafka topics using a JavaScript client.

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.

Subscribe Now

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.

Recent Newsletters