Course: Kafka Connect 101

Running Kafka Connect in Docker

5 min
Tim BerglundSr. Director, Developer Advocacy (Course Presenter)
Robin MoffattStaff Developer Advocate (Course Author)

Running Kafka Connect with Docker

You can run a Kafka Connect worker directly as a JVM process on a virtual machine or bare metal, but you might prefer the convenience of running it in a container, using a technology like Kubernetes or Docker. Note that containerized Connect via Docker will be used for many of the examples in this series.

Confluent's Kafka Connect Docker Image

Confluent maintains its own image for Kafka Connect, cp-kafka-connect-base, which provides a basic Connect worker to which you can add your desired JAR files for sink and source connectors, single message transforms, and converters.

Adding JARs to the Base Image with Confluent Hub

You can use Confluent Hub to add your desired JARs. You can accomplish this in two ways, either by installing them at runtime or by creating a new Docker image.

confluent-hub-jar

Adding JARs to the Kafka Connect Docker Image at Runtime

Adding your dependencies at runtime means that you don’t have to create a new image, but it does increase installation time each time your container is run, and it also requires an internet connection. It’s a good option for prototyping work but probably not for a production deployment (for that, see below).

Your JARs should be in a location that causes them to be class loadable by the Connect process, and you’ll need to add an environmental variable that identifies their location (note that in production you will likely have many more environmental variables than just this one). Also make sure to specify the correct version of the Connect base image. Finally, you should add a command that overrides the base image’s default command so that you can call the Confluent Hub utility, which will install the connectors specified (in this case, the Neo4j connector).

---
version: '2'
services:
  kafka-connect:
    image: confluentinc/cp-kafka-connect-base:6.2.0
    environment:
      []

    command: 
      - bash 
      - -c 
      - |
        confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.9
        /etc/confluent/docker/run

Creating a New Kafka Connect Image with Additional JARs Baked In

The second way to add dependencies to the base image, and the option probably most often used in production deployments, is to build a new image:

FROM confluentinc/cp-kafka-connect-base:6.2.0

RUN confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.9

Make sure to use the correct Confluent base image version and also check the specific documentation for each of your connectors.

Creating Connectors When the Container Launches

Typically, you will create connectors once the worker process is running by manually submitting the configuration or via an external automation. However, you may find—perhaps for demo purposes—that you want a self-sufficient container that also creates the connector when it starts. To do this, you can use a launch script that looks like this:

# Launch Kafka Connect
/etc/confluent/docker/run &
#
# Wait for Kafka Connect listener
echo "Waiting for Kafka Connect to start listening on localhost ⏳"
while : ; do
  curl_status=$$(curl -s -o /dev/null -w %{http_code} http://localhost:8083/connectors)
  echo -e $$(date) " Kafka Connect listener HTTP state: " $$curl_status " (waiting for 200)"
  if [ $$curl_status -eq 200 ] ; then
    break
  fi
  sleep 5 
done

echo -e "\n--\n+> Creating Data Generator source"
curl -s -X PUT -H  "Content-Type:application/json" http://localhost:8083/connectors/source-datagen-01/config \
    -d '{
    "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "kafka.topic": "ratings",
    "max.interval":750,
    "quickstart": "ratings",
    "tasks.max": 1
}'
sleep infinity

Use the promo code CONNECT101 to receive $101 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.