June 8, 2021 | Episode 161

Adopting OpenTelemetry in Confluent and Beyond ft. Xavier Léauté

  • Transcript
  • Notes

Collecting internal, operational telemetry from Confluent Cloud services and thousands of clusters is no small feat. Stakeholders need to rely on the same data to make operational decisions. Whether it be metrics from clusters in Confluent Cloud or traces from our internal service, they all provide valuable insights not only to engineering teams but also to customers for their own operations and for business reporting needs. Traditionally, this data needs to be collected in multiple ways to satisfy all the different requirements. We leverage third-party vendors for our operational needs, which usually means deploying vendor agents or libraries in addition to our own, as we also need to collect some of the same data to expose to customers.

However, this sometimes leads to discrepancies between various systems, which are often hard to reconcile and make it harder to troubleshoot issues across engineering, data science, and other teams.

One of the earliest software engineers at Confluent, Xavier Léauté is no stranger to this. At Confluent, he leads our observability engineering efforts in Confluent Cloud.

With OpenTelemetry, we can collect data in a vendor-agnostic way. It defines a standard format that all our services can use to expose telemetry, and it provides Go and Java libraries that we can use to instrument our services. Many vendors already integrate with OpenTelemetry, which gives us the flexibility to try out different observability solutions with minimal effort, without the need to rewrite applications or deploy new agents. This means that the same data we send to third parties can also be collected internally (in our own clusters).

The same source of data can then be leveraged in many different ways:

  • Using Kafka Connect, we can send this data to our data warehouse and data science teams in real time to derive many of the metrics that we use to track the health of our cloud business
  • That very same data also powers our Cloud Metrics API to provide our customers visibility into their infrastructure
  • Engineers and support teams can collect more fine-grained data to troubleshoot incidents or understand low-level application behavior

We’ve also adopted the same approach for on-prem customers, which enables us to collect telemetry into our cloud and help them troubleshoot issues, leveraging the same infrastructure that we already built for Cloud. 

Regarding OpenTelemetry efforts in Apache Kafka®, we’re working on KIP-714 which will allow us to collect Kafka client metrics to help better understand client-side problems without the need to instrument client applications. Our ultimate goal has always been to migrate to OpenTelemetry, which is now underway. We’d like to make a way for direct integration with OpenTelemetry in Kafka, based on the work that we’ve done at Confluent.

Continue Listening

Episode 162June 10, 2021 | 9 min

Confluent Platform 6.2 | What’s New in This Release + Updates

Based on Apache Kafka® 2.8, Confluent Platform 6.2 introduces Health+, which offers intelligent alerting, cloud-based monitoring tools, and accelerated support so that you can get notified of potential issues before they manifest as critical problems that lead to downtime and business disruption.

Episode 163June 15, 2021 | 25 min

Boosting Security for Apache Kafka with Confluent Cloud Private Link ft. Dan LaMotte

Enabling private links on the cloud is increasingly important for security across networks and even the reliability of stream processing. Staff Software Engineer II Dan LaMotte and his team focus on making secure connections for customers to utilize Confluent Cloud. With the option of private links, you can now also build microservices that use new functionality that wasn’t available in the past. You no longer need to segment your workflow, thanks to completely secure connections between teams that are otherwise disconnected from one another.

Episode 164June 22, 2021 | 35 min

Chaos Engineering with Apache Kafka and Gremlin

The most secure clusters aren’t built on the hopes that they’ll never break. They are the clusters that are broken on purpose and with a specific goal. When organizations want to avoid systematic weaknesses, chaos engineering with Apache Kafka® is the route to go. Patrick Brennan (Principal Architect) and Tammy Butow (Principal SRE) from Gremlin discuss how they do their own chaos engineering to manage and resolve high-severity incidents across the company.

Got questions?

If there's something you want to know about Apache Kafka, Confluent or event streaming, please send us an email with your question and we'll hope to answer it on the next episode of Ask Confluent.

Email Us

Never miss an episode!

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.