course: Kafka Connect 101

Hands On: Getting Started with Kafka Connect

5 min
Danica Fine

Danica Fine

Senior Developer Advocate (Presenter)

Hands On: Getting Started with Kafka Connect

In this exercise, you will create a new topic to hold order event data, and set up a Kafka Connect data generator to populate the topic with sample data.

Confluent Cloud offers dozens of pre-built, fully managed connectors. Use the Amazon CloudWatch Logs or Oracle Database source connectors (among many others!) to stream data into Apache Kafka® or choose from a number of sink connectors to help you move your data into a variety of systems, including BigQuery and Amazon S3. Leveraging these managed connectors is the easiest way to use Kafka Connect to build fully managed data pipelines.

Confluent Cloud

For this course, you will use Confluent Cloud to provide a managed Kafka service, connectors, and stream processing.

  1. Go to Confluent Cloud and create a Confluent Cloud account if you don’t already have one. Otherwise, be sure to log in.
  2. Create a new cluster in Confluent Cloud. For the purposes of this exercise, we’ll be using all of the default configurations for our cluster and choose the Standard cluster. Name the cluster dev.
  3. When you reach the billing information, you’ll notice that you are receiving $400 of free Confluent Cloud usage to start you off. That said, you can use the promo code CONNECT101 for an additional $101 of free usage. Also, when you’re finished with this exercise, don’t forget to delete your connector, topic, and cluster in order to avoid exhausting your free usage and being charged.

Create the orders Topic

From the Topics page of your Confluent Cloud cluster, click on Add topic.

Name the topic orders and ensure that the "Number of partitions" is set to "6."

new-topic-orders

Click on Create with defaults.

Create a Data Generator with Kafka Connect

In reality, the orders topic would probably be populated from an application using the producer API to write messages to it. Here, we’re going to use a data generator that’s available as a connector for Kafka Connect.

  1. In Confluent Cloud, go to your cluster’s "Connectors" page. In the search box, enter datagen.

datagen-source-connector

Select the **Datagen Source Connector**.
  1. Under "Kafka Cluster credentials," click on Generate Kafka API key & secret. Make a note of the generated key and secret as you’ll need these later.

create-an-api-key-datagen

  1. Set the remainder of the options as shown below.

    Which topic do you want to send data to?

    Topic name

    `orders` (as created in the step above)

    Output messages

    Output message format

    JSON

    Datagen Details

    Quick start

    ORDERS

    Max interval between messagse (ms)

    1,000

    Number of tasks for this connector

    Tasks

    1

  2. Click Next. On the confirmation screen, the JSON should look like this:

     {
       "name": "DatagenSourceConnector_0",
       "config": {
         "connector.class": "DatagenSource",
         "name": "DatagenSourceConnector_0",
         "kafka.api.key": "****************",
         "kafka.api.secret": "***********************",
         "kafka.topic": "orders",
         "output.data.format": "JSON",
         "quickstart": "ORDERS",
         "max.interval": "1000",
         "tasks.max": "1"
       }
     }
     
  3. If it doesn’t, return to the previous screen and amend the values as needed. Click Launch to instantiate the connector. This will take a few moments.

  4. On the "Connectors" page of your cluster, you should see the new connector listed. Initially, it will show as “Provisioning,” but in a moment or two, it will switch to “Running.”

connectors-list-data-gen

  1. From the "Topics" page of your cluster, select the orders topic and then Messages. You should see a steady stream of new messages arriving:

new-messages-arriving-on-the-orders-topic

  1. Keep in mind that this Datagen Source Connector is only a jumping off point for your Kafka Connect journey. As a final step, head on over to the “Connectors” page and take a look at the other connectors.

connectors-page-for-kafka-connect

Use the promo code CONNECT101 to receive $101 of free Confluent Cloud usage

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.