Course: Kafka Connect 101

Metrics and Monitoring for Kafka Connect with Confluent

4 min
Tim BerglundSr. Director, Developer Advocacy (Course Presenter)
Robin MoffattStaff Developer Advocate (Course Author)

Metrics and Monitoring for Kafka Connect with Confluent

There are two broad ways to monitor Kafka Connect:

  1. Within Confluent
  2. Monitoring data exposed directly by Kafka Connect, such as JMX and REST.

Monitoring with Confluent

Confluent’s graphical options are the easiest way to get started with monitoring a Connect instance.

Data Flow View

Start with the data flow view as depicted below, which shows where data is coming from and going to. You’ll see sink connectors as consumers on the right, and source connectors as producers on the left.

monitoring-kafka-connect-confluent-data-flow

The data flow view is aware that this is a connector, so you can click on “Inspect” to see more details. In general, for any pipeline, the data flow view is a useful way of seeing what’s going on, whether it depicts producers and consumers, ksqlDB or Kafka Streams, or Connect entities. And of course, its usage gets far more sophisticated than this basic example.

Consumers View

If you click the “Consumers” tab from the data flow view, you’ll find the details of your Connect consumers. Reviewing the image below, you’ll see that four consumers are ksqlDB queries, but the one in the middle with the name connect-lcc-gxj93 is a Connect task.

monitoring-kafka-connect-confluent-data-flow-consumer-groups

Click on that task to drill down to a detailed consumer view, where you’ll be able to see the consumption information for the topic and the offset within each partition of the data consumed so far.

monitoring-kafka-connect-confluent-data-flow-consumer-groups2

Monitoring Using Data Directly Exposed by Kafka Connect

Kafka Connect exposes various data for monitoring over JMX and REST, and this collection is ever expanding (see, for example, KIP-475). These are lower-level options than the Confluent tools shown above. To use JMX, you just need to be familiar with the MBeans that are exposed, and you need a tool for gathering data from them and visualizing it.

In the JMX-based setup in the image below, we see data for total messages read by a sink connector, as well as information about Connect error totals, source records read, and dead letter queue requests. With a little bit of tooling, you can build alerting on this data and expand your own observability framework. All of this setup can be a lot of work, though, so if you can do it in a fully managed way, it is far easier.

kafka-connect-sink-monitoring

Kafka Connect also exposes information about the status of tasks and connectors on its REST interface, which you can read about in the previous module.

Use the promo code CONNECT101 to receive $101 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.