Troubleshooting Confluent Managed Connectors

Troubleshooting Managed Connectors

There are several tools that you can use to troubleshoot managed connectors. Depending upon the type of problem, one of these tools may work better than others. Let’s now walk through a scenario taking a look at how each of these tools can be used to troubleshoot the problem.

Troubleshoot the Dead Letter Queue

One problem that you may experience is a sink connector is unable to process the messages from the Kafka topic it is configured to consume from. It could be that it is just a subset of these messages that it is unable to process or it could be all messages from the topic as shown in this example. Depending upon how the connector is configured, To troubleshoot this, you can click the dead letter queue tile in the connector overview window. This will navigate the UI to the associated Kafka topic where the dead letter queue messages are being written.

Dead Letter Queue Message Header

In the dead letter queue topic view, select the messages tab, drill into one of the messages and select the header tab. Then scroll down in the header information to identify the possible cause for the message ending up in the dead letter queue. In this example, we see that the connector wasn’t configured to auto-create the destination table if it didn’t already exist. To correct this, you would simply updated the connector configuration setting auto create table to true. You could do so using either the Confluent Cloud UI, the Connect API, or the Confluent CLI.

Troubleshoot a Failed Connector

Let’s continue with the use case from the previous slide. The connector has been configured to auto-create the destination table in the MySql database and the Confluent Cloud UI now indicates the connector failed. You can investigate this using several tools. Let’s now look at each of these.

Confluent CLI

The Confluent CLI is one of the tools that can be used to investigate connector failures. The describe command provides similar detail as the Confluent Cloud UI.

Confluent Connect API

There are several tools that you can use to troubleshoot managed connectors. Depending upon the type of problem, one of these tools may work better than others. Let’s now walk through a scenario taking a look at how each of these tools can be used to troubleshoot the problem.

Confluent Cloud Connect Log Events

Confluent Cloud Connect log events are available on the connector events tab. They may provide additional detail regarding connector problems. In this example, the log event adds to the previous trace information about ensuring your input events are a flat struct of primitive fields.

Consuming Connect Log Events

Connect log events can also be accessed using the Confluent CLI consume command. Detailed information regarding how to accomplish this can be found by clicking the triple bar icon in the upper right corner of the Confluent Cloud UI and choosing the Connect log events menu.