Metrics and Monitoring for Kafka Connect with Confluent

Metrics and Monitoring for Kafka Connect with Confluent

There are two broad ways to monitor Kafka Connect:

1. Within Confluent
2. Monitoring data exposed directly by Kafka Connect, such as JMX and REST.

Monitoring with Confluent

Confluent’s graphical options are the easiest way to get started with monitoring a Connect instance.

Data Flow View

Start with the data flow view as depicted below, which shows where data is coming from and going to. You’ll see sink connectors as consumers on the right, and source connectors as producers on the left.

The data flow view is aware that this is a connector, so you can click on “Inspect” to see more details. In general, for any pipeline, the data flow view is a useful way of seeing what’s going on, whether it depicts producers and consumers, ksqlDB or Kafka Streams, or Connect entities. And of course, its usage gets far more sophisticated than this basic example.

Consumers View

If you click the “Consumers” tab from the data flow view, you’ll find the details of your Connect consumers. Reviewing the image below, you’ll see that four consumers are ksqlDB queries, but the one in the middle with the name connect-lcc-gxj93 is a Connect task.

Click on that task to drill down to a detailed consumer view, where you’ll be able to see the consumption information for the topic and the offset within each partition of the data consumed so far.

Monitoring Using Data Directly Exposed by Kafka Connect

Kafka Connect exposes various data for monitoring over JMX and REST, and this collection is ever expanding (see, for example, KIP-475). These are lower-level options than the Confluent tools shown above. To use JMX, you just need to be familiar with the MBeans that are exposed, and you need a tool for gathering data from them and visualizing it.

In the JMX-based setup in the image below, we see data for total messages read by a sink connector, as well as information about Connect error totals, source records read, and dead letter queue requests. With a little bit of tooling, you can build alerting on this data and expand your own observability framework. All of this setup can be a lot of work, though, so if you can do it in a fully managed way, it is far easier.

Kafka Connect also exposes information about the status of tasks and connectors on its REST interface, which you can read about in the previous module.