Senior Developer Advocate (Presenter)
In this exercise, you will create a new topic to hold order event data, and set up a Kafka Connect data generator to populate the topic with sample data.
Confluent Cloud offers dozens of pre-built, fully managed connectors. Use the Amazon CloudWatch Logs or Oracle Database source connectors (among many others!) to stream data into Apache Kafka® or choose from a number of sink connectors to help you move your data into a variety of systems, including BigQuery and Amazon S3. Leveraging these managed connectors is the easiest way to use Kafka Connect to build fully managed data pipelines.
For this course, you will use Confluent Cloud to provide a managed Kafka service, connectors, and stream processing.
dev
.CONNECT101
for an additional $101 of free usage. Also, when you’re finished with this exercise, don’t forget to delete your connector, topic, and cluster in order to avoid exhausting your free usage and being charged.orders
TopicFrom the Topics page of your Confluent Cloud cluster, click on Add topic.
Name the topic orders
and ensure that the "Number of partitions" is set to "6."
Click on Create with defaults.
In reality, the orders
topic would probably be populated from an application using the producer API to write messages to it. Here, we’re going to use a data generator that’s available as a connector for Kafka Connect.
datagen
.Select the **Datagen Source Connector**.
Set the remainder of the options as shown below.
Which topic do you want to send data to? | |
Topic name |
|
Output messages | |
Output message format | JSON |
Datagen Details | |
Quick start | ORDERS |
Max interval between messagse (ms) | 1,000 |
Number of tasks for this connector | |
Tasks | 1 |
Click Next. On the confirmation screen, the JSON should look like this:
{ "name": "DatagenSourceConnector_0", "config": { "connector.class": "DatagenSource", "name": "DatagenSourceConnector_0", "kafka.api.key": "****************", "kafka.api.secret": "***********************", "kafka.topic": "orders", "output.data.format": "JSON", "quickstart": "ORDERS", "max.interval": "1000", "tasks.max": "1" } }
If it doesn’t, return to the previous screen and amend the values as needed. Click Launch to instantiate the connector. This will take a few moments.
On the "Connectors" page of your cluster, you should see the new connector listed. Initially, it will show as “Provisioning,” but in a moment or two, it will switch to “Running.”
orders
topic and then Messages. You should see a steady stream of new messages arriving:We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.