Streaming Audio: A Confluent Podcast about Apache Kafka

You already know Apache Kafka is a distributed event streaming system for setting your data in motion, but how does its internal architecture work? No one can explain the internal architecture better than Jun Rao, one of the original Apache Kafka Creators and Co-Founder of Confluent. Jun has an in-depth understanding of Kafka that few others can claim—and he shares that with us in this episode, and in his new Kafka Internals course on Confluent Developer.

In this episode, Bobby Calderwood, founder of Evident Systems and creator of oNote explains event modeling—a converse approach to the reductive data model system. Event model system is enabled by tools like Apache Kafka, which effectively saves every bit of activity generated by the data system.

New Relic runs one of the larger Apache Kafka installations in the world, ingesting circa 125 petabytes a month, or approximately three billion data points per minute. Anton Rodriguez is the chief architect of the system, responsible for hundreds of clusters and thousands of clients, some of them implemented in non-standard technologies. In addition to the large volume of servers, he works with many teams, which must all work together when issues arise.

Confluent Platform 7.1 expands upon its already innovative features, adding improvements in key areas that benefit data consistency, allow for increased speed and scale, and enhance resilience and reliability. Following the standard for every Confluent release, Confluent Platform 7.1 is built on the most recent version of Apache Kafka 3.1, including KIP-768: extend SASL/OAUTHBEARER with support for OIDC, KIP-773: Differentiate consistently metric latency measured in mills and nanos, as well as KIP-775: Custom partitioners in foreign-key-joins.

Scaling Apache Kafka can be tricky, let alone scaling a team. When he was first hired, Domenico Fioravanti of Therapie Clinic was given the challenging task of assembling a sizable tech team from scratch, while simultaneously building a scalable and decoupled architecture from the ground up. In addition, he wanted to deliver value to the company from day one. One way that Domenico ultimately accomplished these goals was by focusing on managed solutions in order to avoid large investments in engineering know-how. Another way was to deliver quickly to production by using the existing knowledge of his team.