Happy first day of June! And welcome back to your #1 source for up-to-date Apache Kafka® resources. This week, we’re highlighting some classics: the definitive guide to Kafka, Mitch Seymour’s book about Kafka Streams, and Kafka Internals. Remember, if you’ve been working on something Kafka-related, feel free to submit your own resource by emailing info@confluent.io!
You’ve got a realtime data source with Kafka; Flink is a natural ecosystemic counterpart enabling the analysis of that data. If you’re interested in learning more about it, we’ve got you covered:
A few weeks ago, we discussed the history command. Did you know that you can use !{history_id} to run a command with that id from history? So if your history shows:
❯ history -2
5467 ls
5468 cat LICENSE.md
You can run:
❯ !5467
Docs
Downloads
Library
Many thanks to Joseph Morais for this one!
Registration for Current 2023 is now open! There’s more than 100 sessions at Current, all focused on realtime data streaming. Join us, we’d love to have you! If you have specific (or general, we’re not picky) questions about Kafka, feel free to join our Confluent Community Slack and ask away!
We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, send us an email at info@confluent.io!
If you’d like to view previous editions of the newsletter, visit our archive.
If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by scrolling down to the bottom of the page and filling out the form.
P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.
May 18, 2023
We’ve just wrapped up Kafka Summit London… and what an event it was! We met new friends and connected with old ones. We learned from so many folks in the Kafka community. Confluent shared news about Kafka and Confluent Cloud, and also described the upcoming Managed Service for Apache Flink®. If you weren’t able to attend, we’ll be making the talks available online; meanwhile, you can watch the talks from KSL 2022 and Current 2022.
Visit the official Kafka Summit site to watch the keynote and learn more. You can also get started learning Flink with the resources listed below.
Say I have a file, sed_example.txt, with the following contents:
Cat
Cat
Cat
Dog
Dog
Cat
What if I wanted to replace each ‘Cat’ line with the word ‘Dog’? I could use sed, which is a stream editor. Here’s an example of the syntax I’d use:
❯ sed 's/Cat/Dog/' sed_example.txt
Dog
Dog
Dog
Dog
Dog
Dog
Try it for yourself! (Many thanks to Jeff Bean from Confluent for the suggestion!)! We’ll get into more advanced sed tips in future newsletters.
Join the Apacha Kafka meetup in Rome, May 25th! Presentations will be in Italian.
Registration for Current 2023 is now open!
We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or want to submit your own resource for consideration, send us an email at info@confluent.io!
If you’d like to view previous editions of the newsletter, visit our archive.
P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.
May 4, 2023
May the 4th be with you! (Pardon us, we couldn’t resist a Star Wars joke there!) This week, we’ve got a brand new book on data mesh architecture, a new course for .NET developers, and a fun way to look at git internals using Linus Torvalds’ initial commit!
Today’s tip is a quick one. Instead of using the up arrow to tab through your Linux history to find that one extra-complicated command you ran recently, you can run:
history -5
Then your last -5 (or however many you want) commands will show up on the screen! Here’s an example from my computer:
5461 sudo powermetrics --samplers smc |\ngrep -i "CPU die temperature"\n
5462 powermetrics -h
5463 history -10
5464 history -5
5465 history -1
We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or want to submit your own resource for consideration, send us an email at info@confluent.io!
If you’d like to view previous editions of the newsletter, visit our archive.
P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.
April 20, 2023
Well folks, we hope this April is turning out to be a gentle one for you all, with no allergy attacks, whether you’re in the midst of spring or fall. We’ve got a solid roundup of resources today, with some new demos to try, an OCTO-log post on data contracts, and a couple of snappy Kafka terminal aliases. Enjoy!
First on the menu… a refreshed suite of demos for you! There’s a good range here – you can use these demos to cement the foundational concepts of Kafka in your learning path, or you can use the more advanced ones to evaluate the business value of adding Kafka to your data pipelines. We also found some ready-to-use sources of real-world real-time data to power your own demos and explorations.
Uniting the Machine Learning and Data Streaming Ecosystems: Pt 1 This first installment in a series by Robbie Palmer looks at the state of these two ecosystems, and the associated barriers and opportunities in their overlap.
The latest from the OCTO-log: Jack Vanlightly explores the depth of the concept of data contracts for tech organizations.
Unknown Magic Byte! If this error ever struck terror into your heart, read on…
Today’s tip is about aliases that make it a lot easier to work with Kafka. They come from our community member Neil Buesing, who has also written a blog post touching on the topic. The first alias saves you from having to type out the bootstrap server options every time, and it also makes sure that the output is printed in bytes:
alias kcc='kafka-console-consumer \
--bootstrap-server localhost:19092 \
--key-deserializer=org.apache.kafka.common.serialization.BytesDeserializer \
--property print.key=true \
--property key.separator="|" \
--from-beginning \
--topic'
The second alias saves you the work of typing out the bootstrap server options when you’re working with a local container deployment of kafka:
alias kt='kafka-topics --bootstrap-server localhost:19092,localhost:29092,localhost:39092'
The Philosophize This! podcast gives entertaining, high-level summaries of various philosopher’s contributions to the world.
You’ve heard of failing DNS errors taking down massively popular websites for hours. Maybe you know exactly what that entails and you’d like to practice with DNS to avoid similar scenarios, or maybe you just want to learn what the heck DNS is. Mess With DNS is a cool resource from Julia Evans that can meet your needs.
You’ve heard of Wordle, but can you guess a word using… semantics?
Kafka Summit London is happening soon, May 16-17! Register at the conference home page.
Join Kafka developers in-person for a meetup in Atlanta! Other upcoming opportunities to network and learn are
If you’d like to view previous editions of the newsletter, visit Confluent Developer.
P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.
April 6, 2023
We’ve got a veritable treasure trove of resources today! Francesco Tisiot explains the vital issue of message ordering, Michael Drogalis discusses GraalVM as a possible leveler when it comes to stream processing in your language of choice, and we’ve also got an intro to jq. Read on…
This one is back-to-basics for some, but for anyone unfamiliar with jq, it’s an introductory command. A curl request to the mock API, JSONplaceholder, returns an object:
curl -X GET https://jsonplaceholder.typicode.com/posts/1
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}
What if we wanted only the body of that object? Jq to the rescue! We can use its Object-Identifier Index, signified by a dot before the name of the object, to filter out the rest of the data:
❯ curl -X GET https://jsonplaceholder.typicode.com/posts/1 | jq .body
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 292 100 292 0 0 2554 0 --:--:-- --:--:-- --:--:-- 2561
"quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
There’s much more you can do with jq, but you can already see how useful it is for filtering terminal output.
Kafka Summit London is happening soon, May 16-17! Register at the conference home page.
Join Kafka developers in-person for a meetup in Atlanta! Other upcoming opportunities to network and learn are
If you’d like to view previous editions of the newsletter, visit our archive.
P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.
March 23rd, 2023
Hello everyone and welcome to our second edition of the Confluent DevX Newsletter! If you happen to live in participating states in the U.S., we hope daylight savings wasn’t too rough. Wherever you live, we hope that your Pi Day was filled with mathematical joy. We’ve got all sorts of goodies this week ranging from Apache Kafka® migration tips to animals crocheted with patterns devised by ChatGPT. So let’s dive in…
You might have heard of the jq utility to transform JSON data, but what about XML? Enter python-yq. Installing it allows you to use xq, which formats XML to JSON like so:
curl --show-error --silent \
https://tfl.gov.uk/tfl/syndication/feeds/cycle-hire/livecyclehireupdates.xml \ | xq -c '.stations.station[] + {lastUpdate: .stations."@lastUpdate"}'
Result:
{"id":"850","name":"Brandon Street, Walworth","terminalName":"300060","lat":"51.489102","long":"-0.0915489","installed":"true","locked":"false","installDate":null,"removalDate":null,"temporary":"false","nbBikes":"11","nbStandardBikes":"10","nbEBikes":"1","nbEmptyDocks":"11","nbDocks":"22","lastUpdate":"1677702601270"}
Here you see that bicycle rental data, expressed as XML, is transformed into JSON objects by the xq utility. Neat!
Kafka Summit London is happening soon, May 16-17! Register at the conference home page.
March 9th, 2023
Welcome to our inaugural edition of “Confluent DevX Newsletter”! We’ll share many types of content in this newsletter, including: Kafka-related content from the community, tips and tricks for managing Apache Kafka® projects in your terminal, and links to fun nerdy content from around the broader web.
To view the output of a command in bytes, use a pipe and ‘hexdump.’ For example, if I were to want the output of viewing a topic using kcat to be in bytes, I’d type:
❯ kcat -Ct purchases -c1 | hexdump
0000000 0000 8601 00a1 0308 0612 7567 7469 7261
By the way…
We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback on what type of content you’d like to see, or you want to submit your own resource for consideration, send us an email at info@confluent.io!
P.S. If you want to learn more about Kafka or Confluent Cloud, visit our developer site at Confluent Developer.
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.