Running Kafka Connect

Connectors

When running Kafka Connect, instances of connector plugins provide the integration between external data systems and the Kafka Connect framework. These connector plugins are reusable components that define how source connectors ought to capture data from data sources to a Kafka topic and also how sink connectors should copy data from Kafka topics to be recognized by a target system. By taking care of all of this boilerplate logic for you, the plugins allow you to hit the ground running with Kafka Connect and focus on your data.

There are hundreds of connector plugins available for a variety of data sources and sinks. There are dozens of fully managed connectors available for you to run entirely through Confluent Cloud. Plus, connectors can also be downloaded from Confluent Hub for use with self-managed Kafka Connect.

Let’s dive a little bit more into the fully managed and self-managed connectors and what those mean to you.

Confluent Cloud Managed Connectors

Confluent Cloud offers pre-built, fully managed, Apache Kafka connectors that make it easy to instantly connect to popular data sources and sinks. With a simple UI-based configuration and elastic scaling with no infrastructure to manage, Confluent Cloud connectors make moving data in and out of Kafka an effortless task, giving you more time to focus on application development.

To start, you simply select the connector and fill in a few configuration details about your source or target system. This can be done using the Confluent Cloud console, the Confluent CLI, or the Confluent Connect API.

From there, Confluent takes care of the rest on your behalf:

• Using the configuration settings you specified, your connector instance is provisioned and run
• The execution of the connector instance is monitored
• Should the connector fail, you’ll have access to troubleshooting to help identify the root cause, correct the issue, and restart the connector and its tasks

All in all, you can relax knowing that all of these tasks are being handled for you.

That said, there are a few limitations regarding managed connectors:

• Some self-managed connectors that are available on Confluent Hub for installation in self-managed Kafka Connect clusters are not yet available in Confluent Cloud
• Some fully managed Confluent Cloud connectors are not available for all cloud providers
• Some configuration settings available for self-managed connectors may not be available for Confluent managed connectors
• Some single message transformations (SMTs) that are available for use in self-managed Kafka Connect clusters are not available in Confluent Cloud

Be sure to keep those things in mind as you choose which connector options are best for you.

Self-Managed Kafka Connect

So long as you have access to a Kafka cluster, Kafka Connect can also be run as a self-managed Kafka Connect cluster, but as you can see from the diagram, there is a lot more involved with doing so:

• Self-managed Kafka Connect consists of one or more Connect clusters depending upon the requirement
• Each cluster consists of one or more Connect worker machines on which the individual connector instances run

Regardless of how you choose to run Kafka Connect, it’s helpful to understand the individual Kafka Connect components and how they work together.

Kafka Connect Workers

Ultimately, Kafka Connect workers are just JVM processes that you can deploy on bare metal or containers.

A few options present themselves:

• You’re free to run a bare-metal, on-premises install of Confluent Platform
• For those leveraging infrastructure as a service, you may install Confluent Platform on those resources
• Terraform is an option on a couple cloud providers
• And of course, there’s Docker which you can use for both on-prem and cloud-based installations

Managing a Kafka Connect Cluster

Once your Kafka Connect cluster is up and running, there’s a bit of management that needs to be done:

• Connect workers have a number of default configuration settings that you may need to alter
• Depending on the needs of your systems, you might need to scale the Connect cluster up or down to suit demand changes
• And of course, you’ll be monitoring for problems and fixing those that occur