Senior Developer Advocate (Presenter)
In this exercise, you will create a new topic to hold order event data, and set up a Kafka Connect data generator to populate the topic with sample data.
Confluent Cloud offers dozens of pre-built, fully managed connectors. Use the Amazon CloudWatch Logs or Oracle Database source connectors (among many others!) to stream data into Apache Kafka® or choose from a number of sink connectors to help you move your data into a variety of systems, including BigQuery and Amazon S3. Leveraging these managed connectors is the easiest way to use Kafka Connect to build fully managed data pipelines.
For this course, you will use Confluent Cloud to provide a managed Kafka service, connectors, and stream processing.
kc-101
.kc-101
cluster since we will be using it in other exercises for this course.From the Topics page of your Confluent Cloud cluster, click on Add topic.
Name the topic orders
and ensure that the Number of partitions is set to 6
.
In reality, our Kafka topic would probably be populated from an application using the producer API to write messages to it. Here, we’re going to use a data generator that’s available as a connector for Kafka Connect.
datagen
.
Select the Datagen Source connector.
orders
and ensure that the Number of partitions is set to 6
.There is no need to save them as we will not be using them after this exercise.
{
"name": "DatagenSourceConnector_0",
"config": {
"connector.class": "DatagenSource",
"name": "DatagenSourceConnector_0",
"kafka.auth.mode": "KAFKA_API_KEY",
"kafka.api.key": "****************",
"kafka.api.secret": "****************************************************************",
"kafka.topic": "orders",
"output.data.format": "JSON",
"quickstart": "ORDERS",
"tasks.max": "1"
}
}
18. From the Topics page of your cluster, select the
orders
topic and then Messages. You should see a steady stream of new messages arriving:
19. Keep in mind that this Datagen Source Connector is only a jumping off point for your Kafka Connect journey. As a final step, head on over to the Connectors page and take a look at the other connectors.
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.