When running Kafka Connect, instances of connector plugins provide the integration between external data systems and the Kafka Connect framework. These connector plugins are reusable components that define how source connectors ought to capture data from data sources to a Kafka topic and also how sink connectors should copy data from Kafka topics to be recognized by a target system. By taking care of all of this boilerplate logic for you, the plugins allow you to hit the ground running with Kafka Connect and focus on your data.
There are hundreds of connector plugins available for a variety of data sources and sinks. There are dozens of fully managed connectors available for you to run entirely through Confluent Cloud. Plus, connectors can also be downloaded from Confluent Hub for use with self-managed Kafka Connect.
Let’s dive a little bit more into the fully managed and self-managed connectors and what those mean to you.
Confluent Cloud offers pre-built, fully managed, Apache Kafka connectors that make it easy to instantly connect to popular data sources and sinks. With a simple UI-based configuration and elastic scaling with no infrastructure to manage, Confluent Cloud connectors make moving data in and out of Kafka an effortless task, giving you more time to focus on application development.
To start, you simply select the connector and fill in a few configuration details about your source or target system. This can be done using the Confluent Cloud console, the Confluent CLI, or the Confluent Connect API.
From there, Confluent takes care of the rest on your behalf:
All in all, you can relax knowing that all of these tasks are being handled for you.
That said, there are a few limitations regarding managed connectors:
Be sure to keep those things in mind as you choose which connector options are best for you.
So long as you have access to a Kafka cluster, Kafka Connect can also be run as a self-managed Kafka Connect cluster, but as you can see from the diagram, there is a lot more involved with doing so:
Regardless of how you choose to run Kafka Connect, it’s helpful to understand the individual Kafka Connect components and how they work together.
Ultimately, Kafka Connect workers are just JVM processes that you can deploy on bare metal or containers.
A few options present themselves:
Once your Kafka Connect cluster is up and running, there’s a bit of management that needs to be done:
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.