April 13, 2022 | Episode 209

Monitoring Extreme-Scale Apache Kafka Using eBPF at New Relic

  • Transcript
  • Notes

New Relic runs one of the larger Apache Kafka® installations in the world, ingesting circa 125 petabytes a month, or approximately three billion data points per minute. Anton Rodriguez is the architect of the system, responsible for hundreds of clusters and thousands of clients, some of them implemented in non-standard technologies. In addition to the large volume of servers, he works with many teams, which must all work together when issues arise.

Monitoring New Relic's large Kafka installation is critical and of course challenging, even for a company that itself specializes in monitoring. Specific obstacles include determining when rebalances are happening, identifying particularly old consumers, measuring consumer lag, and finding a way to observe all producing and consuming applications.

One way that New Relic has improved the monitoring of its architecture is by directly consuming metrics from the Linux kernel using its new eBPF technology, which lets programs run inside the kernel without changing source code or adding additional modules (the open-source tool Pixie enables access to eBPF in a Kafka context). eBPF is very low impact, so doesn’t affect services, and it allows New Relic to see what’s happening at the network level—and to take action as necessary.

Continue Listening

Episode 210April 21, 2022 | 51 min

Using Event-Driven Design with Apache Kafka Streaming Applications ft. Bobby Calderwood

In this episode, Bobby Calderwood, founder of Evident Systems and creator of oNote explains event modeling—a converse approach to the reductive data model system. Event model system is enabled by tools like Apache Kafka, which effectively saves every bit of activity generated by the data system.

Episode 211April 28, 2022 | 48 min

Optimizing Apache Kafka's Internals with Its Co-Creator Jun Rao

You already know Apache Kafka is a distributed event streaming system for setting your data in motion, but how does its internal architecture work? No one can explain the internal architecture better than Jun Rao, one of the original Apache Kafka Creators and Co-Founder of Confluent. Jun has an in-depth understanding of Kafka that few others can claim—and he shares that with us in this episode, and in his new Kafka Internals course on Confluent Developer.

Episode 212May 3, 2022 | 2 min

Build a Data Streaming App with Apache Kafka and JS - Coding in Motion

Coding is inherently enjoyable and experimental. With the goal of bringing fun into programming, Kris Jenkins (Senior Developer Advocate, Confluent) hosts a new series of hands-on workshops—Coding in Motion, to teach you how to use Apache Kafka and data streaming technologies for real-life use cases.

Got questions?

If there's something you want to know about Apache Kafka, Confluent or event streaming, please send us an email with your question and we'll hope to answer it on the next episode of Ask Confluent.

Email Us

Never miss an episode!

Confluent Cloud is a fully managed Apache Kafka service available on all three major clouds. Try it for free today.

Try it for free