Hands On: Getting Started with Kafka Connect

In this exercise, you will create a new topic to hold order event data, and set up a Kafka Connect data generator to populate the topic with sample data.

Confluent Cloud offers dozens of pre-built, fully managed connectors. Use the Amazon CloudWatch Logs or Oracle Database source connectors (among many others!) to stream data into Apache Kafka® or choose from a number of sink connectors to help you move your data into a variety of systems, including BigQuery and Amazon S3. Leveraging these managed connectors is the easiest way to use Kafka Connect to build fully managed data pipelines.

Confluent Cloud

For this course, you will use Confluent Cloud to provide a managed Kafka service, connectors, and stream processing.

1. Go to Confluent Cloud and create a Confluent Cloud account if you don’t already have one. Otherwise, be sure to log in.
2. Create a new cluster in Confluent Cloud. For the purposes of this exercise, we’ll be using all of the default configurations for our cluster and choose the Standard cluster. Name the cluster dev.
3. When you reach the billing information, you’ll notice that you are receiving $400 of free Confluent Cloud usage to start you off. That said, you can use the promo code CONNECT101 for an additional $101 of free usage. Also, when you’re finished with this exercise, don’t forget to delete your connector, topic, and cluster in order to avoid exhausting your free usage and being charged.

Create the orders Topic

From the Topics page of your Confluent Cloud cluster, click on Add topic.

Name the topic orders and ensure that the "Number of partitions" is set to "6."

Click on Create with defaults.

Create a Data Generator with Kafka Connect

In reality, the orders topic would probably be populated from an application using the producer API to write messages to it. Here, we’re going to use a data generator that’s available as a connector for Kafka Connect.

1. In Confluent Cloud, go to your cluster’s "Connectors" page. In the search box, enter datagen.

   Select the Datagen Source Connector.

2. Under "Kafka Cluster credentials," click on Generate Kafka API key & secret. Make a note of the generated key and secret as you’ll need these later.

3. Set the remainder of the options as shown below.

   Which topic do you want to send data to?
   Topic name	`orders` (as created in the step above)
   Output messages
   Output message format	JSON
   Datagen Details
   Quick start	ORDERS
   Max interval between messagse (ms)	1,000
   Number of tasks for this connector
   Tasks	1

4. Click Next. On the confirmation screen, the JSON should look like this:

   {
     "name": "DatagenSourceConnector_0",
     "config": {
       "connector.class": "DatagenSource",
       "name": "DatagenSourceConnector_0",
       "kafka.api.key": "****************",
       "kafka.api.secret": "***********************",
       "kafka.topic": "orders",
       "output.data.format": "JSON",
       "quickstart": "ORDERS",
       "max.interval": "1000",
       "tasks.max": "1"
     }
   }
   

5. If it doesn’t, return to the previous screen and amend the values as needed. Click Launch to instantiate the connector. This will take a few moments.
6. On the "Connectors" page of your cluster, you should see the new connector listed. Initially, it will show as “Provisioning,” but in a moment or two, it will switch to “Running.”
7. From the "Topics" page of your cluster, select the orders topic and then Messages. You should see a steady stream of new messages arriving:
8. Keep in mind that this Datagen Source Connector is only a jumping off point for your Kafka Connect journey. As a final step, head on over to the “Connectors” page and take a look at the other connectors.