2025 - Happy New(sletter) Year!

Dear Data Streaming Community,

Wish you a happy , prosperous and joyful 2025!

We start 2025 afresh with plans to make 2025 a memorable one for the community!

While we look forward to exciting new releases of Apache Kafka® and Apache Flink®, we eagerly wait for Current 2025, the biggest data streaming event with 3 editions (Bengaluru, London, New Orleans!).

So, sit back, relax and wade through upcoming news and plans for the data streaming community in 2025!

Current London 2025: Call for Papers

Current 2025 London, the most popular data streaming event takes place during May 20-21 2025 at ExCel London.

Call for papers for Current London 2025 is open till February 17th, 2025.

If you have an impactful data streaming or data platform story, that you will like to share with the community, submit your session abstracts now!

Current Bengaluru 2025 - Mark your calendar

Mark your calendar for March 19th, 2025 when Current 2025 Bengaluru, the biggest data streaming event of the year hits the road ! Call for paper is over with an unprecedented number of submissions!

Registrations will open shortly, stay tuned for more information!

Data Streaming Resources:

• Are software engineering teams and data engineering really two different teams working on their respective goals? Learn how technology and architecture patterns like data products, data contracts, shift left etc .. have filled this chasm and made both these disciplines come closer together in this blog from Jack Vanlightly, Principal Technologist from Confluent.
• Vu Trinh explores the innermost nuances of WarpStream and dives deep into WarpStream agents, agent groups, service discovery, compaction, message ordering, among other areas. Read his informative blog on why BYOC(Bring Your Own Cloud) and WarpStream makes complete sense for certain streaming use cases.

Links From Around the Web:

• How real is ‘real-time’? Read Zach Wilson’s linkedin post on acceptable latencies for real-time use cases.
• Databases in 2024: A Year in Review - Andy Pavlo’s interesting review on what happened in the database world.
• Read a concise, yet important streaming data best practices from Yaroslav Tkachenko.
• Dive into Kafka's architecture with SoftwareMill’s interactive tool

  • Understanding Kafka for the first time can feel overwhelming. But it doesn’t have to be. SoftwareMill designed an intuitive visual tool to explain how Kafka works.

    This interactive tool breaks down key concepts like brokers, producers, consumers, and partitions with graphical representations that bring them to life. But it doesn’t stop there. It lets you experiment with various settings to see real-time impacts on the entire message-processing flow.

    Adjust the number of brokers and watch the system adapt as they’re turned on or off. Explore different replication factors and their influence on data resilience.
    Simulate increasing loads and discover how Kafka handles the pressure.
    Understand how offsets are committed and their role in message redelivery when consumers are added or removed.

Catalyst Insight:

In our brand-new ‘Catalyst Insight’ section we intend to ask catalysts from the data streaming community to share their experiences.

In this edition, we request Sami Alashabi, Solution Architect at Essent to share his insights.

Sami is an Architect at Essent, with a passion to unlock value from data. His main area of expertise lies within the fields of Big Data Analytics, Stream Processing, Cloud Architecture, Microservices and Integrations. When not architecting or engineering, he likes to travel and enjoying quality time with family.

How would you describe your role in the data world? Not necessarily as in your title, but what unique perspective and experiences do you bring?

“In the data world, I see myself as a strategic enabler and innovator with a strong focus on streaming and event-driven architectures (EDA). I specialize in designing scalable, real-time systems that empower businesses to harness the full potential of their data. My experience includes implementing event streaming platforms, such as Kafka, and driving the adoption of EDA principles across organizations, ensuring seamless integration and alignment with business goals.

I bring a unique combination of technical expertise and strategic vision, enabling me to translate complex data requirements into practical, future-proof solutions. By fostering collaboration and sharing knowledge, I help teams embrace the power of event-driven systems to deliver actionable insights and create value at speed.”

Can you tell us the story of an interesting data streaming bug you ran into and solved at one point?

Context: Early in my career as a streaming engineer and architect, I encountered a challenging issue with Kafka's compact topics. These topics are designed to retain only the latest record for each key, which is useful for maintaining a unique state per key.

The Bug: I mistakenly designed the partition strategy and event key incorrectly. Specifically, I used a key that was not unique enough for different events, causing Kafka to overwrite events. This meant that different events with the same key were being lost, as only the latest event was retained.

Investigation:
Compact Topics: By default, compact topics keep only the latest value for each key.

Event Keys: I had used a partial primary key that did not uniquely identify each event.

Overwriting Issue: This resulted in multiple events being treated as the same, leading to data loss as only the latest event was stored.

Solution: To resolve this, I adjusted the key strategy:

Key-Value Pair: I enhanced the event key to include an additional unique identifier (e.g., a timestamp or UUID). This combination ensured that each event had a unique key.

Unique Primary Key: The new key structure was a composite key combining the initial key and a unique ID, which together formed a unique identifier for each event.

Outcome: This adjustment ensured that each event was uniquely identified, preventing overwriting in the compact topic. As a result, all events were stored correctly without loss, maintaining the integrity of our data stream.

What advice would you offer a burgeoning data streaming engineer?

As a burgeoning data streaming engineer, it's crucial to master the basics of event sourcing, stream processing, and stateful operations, while gaining hands-on experience with tools like Apache Kafka, Flink, and Pulsar. Focus on effective partitioning and continuous optimization to ensure low latency and high throughput. Implement fault tolerance strategies like checkpointing and set up robust monitoring systems to maintain reliability. Embrace the principles of distributed systems by designing for concurrency and managing data consistency. Engage with the community through forums and open-source projects to stay updated and seek advice. Keep learning and experimenting with new ideas to innovate. Develop strong communication skills to collaborate effectively with engineers, archtiects and business stakeholders, ensuring your work integrates seamlessly into broader projects. Embrace the challenges of this dynamic field, and enjoy the rewarding journey of mastering data streaming.

Want to learn more about our Confluent Community Catalyst Program? Visit the page here to get all the details!

Upcoming Events:

In-Person Meetups:

• Pune - Jan 25th
• Gurgaon - Jan 25th
• Mumbai - Jan 25th
• Hamburg - Jan 28th
• Lille - Jan 29th
• Bay Area - Jan 29th
• Coimbatore - Feb 1st
• Stockholm - Feb 5th
• Copenhagen @ Nine A/S - Feb 6th

Stay up to date with all Confluent-run meetup events - by copying the following link into your personal calendar platform:

https://airtable.com/app8KVpxxlmhTbfcL/shrNiipDJkCa2GBW7/iCal

(Instructions for GCal, iCal, Outlook, etc.) 

By the way…

We hope you enjoyed our curated assortment of resources! If you’d like to provide feedback, suggest ideas for content you’d like to see, or you want to submit your own resource for consideration, email us at devx_newsletter@confluent.io!

If you’d like to view previous editions of the newsletter, visit our archive.

If you’re viewing this newsletter online, know that we appreciate your readership and that you can get this newsletter delivered directly to your inbox by filling out the sign-up form on the left-hand side.

P.S. If you want to learn more about Kafka, Flink, or Confluent Cloud, visit our developer site at Confluent Developer.