Get Started Free
newsletter

The Confluent DevX Newsletter

Classic resources on Kafka, Current 2023, and mantis shrimp

Happy first day of June! And welcome back to your #1 source for up-to-date Apache Kafka® resources. This week, we’re highlighting some classics: the definitive guide to Kafka, Mitch Seymour’s book about Kafka Streams, and Kafka Internals. Remember, if you’ve been working on something Kafka-related, feel free to submit your own resource by emailing info@confluent.io!

Apache Kafka Resources:

You’ve got a realtime data source with Kafka; Flink is a natural ecosystemic counterpart enabling the analysis of that data. If you’re interested in learning more about it, we’ve got you covered:

Terminal Tip of the Week:

A few weeks ago, we discussed the history command. Did you know that you can use !{history_id} to run a command with that id from history? So if your history shows:

❯ history -2
 5467  ls
 5468  cat LICENSE.md

You can run:

❯ !5467
Docs
Downloads
Library

Many thanks to Joseph Morais for this one!

In Other News:

Registration for Current 2023 is now open! There’s more than 100 sessions at Current, all focused on realtime data streaming. Join us, we’d love to have you! If you have specific (or general, we’re not picky) questions about Kafka, feel free to join our Confluent Community Slack and ask away!

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, send us an email at info@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by scrolling down to the bottom of the page and filling out the form.

P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.

May 18, 2023

We’ve just wrapped up Kafka Summit London… and what an event it was! We met new friends and connected with old ones. We learned from so many folks in the Kafka community. Confluent shared news about Kafka and Confluent Cloud, and also described the upcoming Managed Service for Apache Flink®. If you weren’t able to attend, we’ll be making the talks available online; meanwhile, you can watch the talks from KSL 2022 and Current 2022.

Visit the official Kafka Summit site to watch the keynote and learn more. You can also get started learning Flink with the resources listed below.

Apache Kafka® Resources:

Terminal Tip of the Week:

Say I have a file, sed_example.txt, with the following contents:

Cat
Cat
Cat
Dog
Dog
Cat

What if I wanted to replace each ‘Cat’ line with the word ‘Dog’? I could use sed, which is a stream editor. Here’s an example of the syntax I’d use:

❯ sed 's/Cat/Dog/' sed_example.txt
Dog
Dog
Dog
Dog
Dog
Dog

Try it for yourself! (Many thanks to Jeff Bean from Confluent for the suggestion!)! We’ll get into more advanced sed tips in future newsletters.

In Other News:

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or want to submit your own resource for consideration, send us an email at info@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.

Build an event-driven data mesh, use .NET with Kafka, & other skills

May 4, 2023

May the 4th be with you! (Pardon us, we couldn’t resist a Star Wars joke there!) This week, we’ve got a brand new book on data mesh architecture, a new course for .NET developers, and a fun way to look at git internals using Linus Torvalds’ initial commit!

Apache Kafka® Resources:

Terminal Tip of the Week:

Today’s tip is a quick one. Instead of using the up arrow to tab through your Linux history to find that one extra-complicated command you ran recently, you can run:

history -5

Then your last -5 (or however many you want) commands will show up on the screen! Here’s an example from my computer:

 5461  sudo powermetrics --samplers smc |\ngrep -i "CPU die temperature"\n
 5462  powermetrics -h
 5463  history -10
 5464  history -5
 5465  history -1

In Other News:

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or want to submit your own resource for consideration, send us an email at info@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.

Confluent Developer DevX Newsletter #4: Data contracts and more Kafka stuff

April 20, 2023

Well folks, we hope this April is turning out to be a gentle one for you all, with no allergy attacks, whether you’re in the midst of spring or fall. We’ve got a solid roundup of resources today, with some new demos to try, an OCTO-log post on data contracts, and a couple of snappy Kafka terminal aliases. Enjoy!

Apache Kafka® Resources:

  • First on the menu… a refreshed suite of demos for you! There’s a good range here – you can use these demos to cement the foundational concepts of Kafka in your learning path, or you can use the more advanced ones to evaluate the business value of adding Kafka to your data pipelines. We also found some ready-to-use sources of real-world real-time data to power your own demos and explorations.

  • Uniting the Machine Learning and Data Streaming Ecosystems: Pt 1 This first installment in a series by Robbie Palmer looks at the state of these two ecosystems, and the associated barriers and opportunities in their overlap.

  • The latest from the OCTO-log: Jack Vanlightly explores the depth of the concept of data contracts for tech organizations.

  • Unknown Magic Byte! If this error ever struck terror into your heart, read on…

Terminal Tip of the Week:

Today’s tip is about aliases that make it a lot easier to work with Kafka. They come from our community member Neil Buesing, who has also written a blog post touching on the topic. The first alias saves you from having to type out the bootstrap server options every time, and it also makes sure that the output is printed in bytes:

alias kcc='kafka-console-consumer \
        --bootstrap-server localhost:19092 \
        --key-deserializer=org.apache.kafka.common.serialization.BytesDeserializer  \
        --property print.key=true \
        --property key.separator="|" \
        --from-beginning \
        --topic'

The second alias saves you the work of typing out the bootstrap server options when you’re working with a local container deployment of kafka:

alias kt='kafka-topics --bootstrap-server localhost:19092,localhost:29092,localhost:39092'

  • The Philosophize This! podcast gives entertaining, high-level summaries of various philosopher’s contributions to the world.

  • You’ve heard of failing DNS errors taking down massively popular websites for hours. Maybe you know exactly what that entails and you’d like to practice with DNS to avoid similar scenarios, or maybe you just want to learn what the heck DNS is. Mess With DNS is a cool resource from Julia Evans that can meet your needs.

  • You’ve heard of Wordle, but can you guess a word using… semantics?

In Other News:

Kafka Summit London is happening soon, May 16-17! Register at the conference home page.

Join Kafka developers in-person for a meetup in Atlanta! Other upcoming opportunities to network and learn are

  • In Person in London on Kafka Streams in FinTech & Microservices KStreams accelerator
  • Join us virtually for a chat on Apache Kafka and Azure Connectors with a hands on workshop

If you’d like to view previous editions of the newsletter, visit Confluent Developer.

P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.

Stream Processing in Your Language of Choice? Springing into Apache Kafka, machine learning, and more with community content for April. 🐣

April 6, 2023

We’ve got a veritable treasure trove of resources today! Francesco Tisiot explains the vital issue of message ordering, Michael Drogalis discusses GraalVM as a possible leveler when it comes to stream processing in your language of choice, and we’ve also got an intro to jq. Read on…

Apache Kafka® Resources:

Terminal Tip of the Week:

This one is back-to-basics for some, but for anyone unfamiliar with jq, it’s an introductory command. A curl request to the mock API, JSONplaceholder, returns an object:

curl -X GET https://jsonplaceholder.typicode.com/posts/1
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

What if we wanted only the body of that object? Jq to the rescue! We can use its Object-Identifier Index, signified by a dot before the name of the object, to filter out the rest of the data:

❯ curl -X GET https://jsonplaceholder.typicode.com/posts/1 | jq .body
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   292  100   292    0     0   2554      0 --:--:-- --:--:-- --:--:--  2561
"quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"

There’s much more you can do with jq, but you can already see how useful it is for filtering terminal output.

In Other News:

Kafka Summit London is happening soon, May 16-17! Register at the conference home page.

Join Kafka developers in-person for a meetup in Atlanta! Other upcoming opportunities to network and learn are

  • In Person in London on Kafka Streams in FinTech & Microservices KStreams accelerator
  • Join us virtually for a chat on Apache Kafka and Azure Connectors with a hands on workshop

If you’d like to view previous editions of the newsletter, visit our archive.

P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.

March Roundup of Kafka Resources, Pi, and other Concerns

March 23rd, 2023

Hello everyone and welcome to our second edition of the Confluent DevX Newsletter! If you happen to live in participating states in the U.S., we hope daylight savings wasn’t too rough. Wherever you live, we hope that your Pi Day was filled with mathematical joy. We’ve got all sorts of goodies this week ranging from Apache Kafka® migration tips to animals crocheted with patterns devised by ChatGPT. So let’s dive in…

Kafka Resources:

Terminal Tip of the Week:

You might have heard of the jq utility to transform JSON data, but what about XML? Enter python-yq. Installing it allows you to use xq, which formats XML to JSON like so:

curl --show-error --silent \
https://tfl.gov.uk/tfl/syndication/feeds/cycle-hire/livecyclehireupdates.xml \                                              | xq -c '.stations.station[] + {lastUpdate: .stations."@lastUpdate"}'

Result:

{"id":"850","name":"Brandon Street, Walworth","terminalName":"300060","lat":"51.489102","long":"-0.0915489","installed":"true","locked":"false","installDate":null,"removalDate":null,"temporary":"false","nbBikes":"11","nbStandardBikes":"10","nbEBikes":"1","nbEmptyDocks":"11","nbDocks":"22","lastUpdate":"1677702601270"}

Here you see that bicycle rental data, expressed as XML, is transformed into JSON objects by the xq utility. Neat!

In Other News:

Kafka Summit London is happening soon, May 16-17! Register at the conference home page.

Introducing the Confluent DevX Newsletter 🎉

March 9th, 2023

Welcome to our inaugural edition of “Confluent DevX Newsletter”! We’ll share many types of content in this newsletter, including: Kafka-related content from the community, tips and tricks for managing Apache Kafka® projects in your terminal, and links to fun nerdy content from around the broader web.

Apache Kafka® Resources:

  • Streaming Audio podcasts this week: Real-Time Data Transformations and Analytics with dbt Labs We loved what Amy said about developer experience!
  • 🦙Alpacas! And Kafka 3.4! Watch Danica (and tons of cute alpacas) explain what’s new.
  • Blog posts: Windowing in Kafka Streams – Windows, they roll, they tumble, they slide… and then there are session windows! See our blog post for more on what distinguishes these types of windows from each other.
  • Kafka for Python Developers. Recently, a course on Kafka for Python Developers was released to Confluent Developer. We highly recommend it if you’re familiar with Python and itching to try Kafka!
  • Read the OCTOlog for in-depth analysis of those data streaming opinions you hold near and dear! Ben Stopford begins by analyzing the nature of opinion itself.

Terminal Tip of the Week:

To view the output of a command in bytes, use a pipe and ‘hexdump.’ For example, if I were to want the output of viewing a topic using kcat to be in bytes, I’d type:

❯ kcat -Ct purchases -c1 | hexdump

0000000 0000 8601 00a1 0308 0612 7567 7469 7261

In Other News:

  • The Current 2023 CFP (Call For Papers) is open! 🎉 We are looking forward to your thoughtful submissions. If you’re looking for feedback, we have office hours available in the #speakers-office-hours channel in the Confluent Community Slack.
  • We enjoyed this blog post by Confluent Community Catalyst Olena Kutsenko on ways to balance your data across Apache Kafka partitions
  • Kafka Summit London is happening soon, May 16-17! Register at the conference home page.

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback on what type of content you’d like to see, or you want to submit your own resource for consideration, send us an email at info@confluent.io!

P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.

Confluent Cloud is a fully managed Apache Kafka service available on all three major clouds. Try it for free today.

Try it for free

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.