Hands On: Your First Kafka Application in 10 Minutes or Less

So now that you know a little bit more about what Apache Kafka is, without wasting any more time, let's take it for a test drive. But first, we'll need a Kafka cluster. Confluent Cloud is a great resource that provisions a Kafka cluster in seconds, allowing you to get started quickly and really hit the ground running. Within the next five minutes, you'll have a ton of practical experience running and interacting with Kafka. So let's jump in. Confluent Cloud is the easiest way to get started with Apache Kafka. To use it, you'll first need to create a Confluent Cloud account. Go to the URL on the screen. There you'll enter your name, email, and password. Be sure to remember these sign in details as you'll need them to access your account later. Click the Start free button and wait to receive a confirmation email in your inbox. The link in the confirmation email will lead you to the next step where you'll be prompted to set up your cluster. You can choose between a basic, standard, or dedicated cluster. Basic and standard clusters are serverless offerings where your free Confluent Cloud usage is only exhausted based on what you use. This is perfect for what we need today. For the exercises in this course, we'll choose the basic cluster. Usage costs will vary with any of these choices, but they are clearly shown at the bottom of the screen. That being said, once we wrap up these exercises, don't forget to stop and delete any resources that you created to avoid exhausting your free usage. Click Review to get one last look at the choices you've made and give your cluster a name, then Launch. It may take a few minutes for your cluster to be provisioned. But that's it, you'll receive an email once your cluster is fully provisioned. In the meantime, let let's go ahead and leverage that promo code that we saw earlier. From Settings, choose Billing and Payment. You'll see here that you have $400 of free Confluent Cloud usage. But if you select the Payment details and contacts tab, you can either add a credit card or choose to enter a promo code. We'll enter KAFKA101 to get an additional $101 of free usage which will be more than enough to get you through these exercises. With our account set up and our Kafka cluster now provisioned, let's create a topic. As you know from the previous module, Kafka is a collection of immutable append-only logs. The primary component of storage in Kafka is called a topic which typically represents a particular data entity. Knowing how to interact with a topic is a crucial first step on your Kafka journey. From the Confluent Cloud landing page, select the Topics tab on the left-hand side of the screen, then choose Create topic. Let's call our topic poems. You'll see why in just a second. The default number of partitions for a topic is six which works well for today's use case, so go ahead and select Create with defaults. You'll then be taken to the Topic overview screen. Here you'll see high level details about the production and consumption of messages from this topic. There aren't any messages the topic for now, so this page isn't very interesting. Let's go ahead and change that. Select Messages to be taken to the message viewing screen. When data is being produced to your topic, this is a great page to reference to see that data flow in real time. From here we can select Produce a new message to this topic. This will open a UI that lets you enter a key and value for a new message. And remember that a message or an event in Kafka is treated as a key value pair. Delete the existing data in the key and value fields and enter one for key. For value, enter a line from a poem that may sound familiar. All that is gold does not glitter. Click Produce to add the message to your topic. Now before we move on, it's useful to know a bit more about partitions. While topics are the primary component for storage in Kafka, these topics are broken down into even smaller components called partitions. When you write a message to a Kafka topic, that message is actually stored in one of the topic's partitions. The partition that a message is routed to is based on the key of that message. So with that in mind, in the Jump to offset field above enter zero as the offset since we're at the beginning of the topic and then select zero partition three. You should see the message that you just wrote. In a similar fashion, we'll add the next few lines of the poem. Remember when messages are produced to a topic along with different keys, the messages may potentially be distributed to different partitions of the topic as partition is determined by key. The four messages that we just wrote will be distributed amongst the six partitions of our topic. Using the Jump to offset field, we can explore the remaining partitions of the topi and see the partition in which these messages were written. Check it out yourself. Producing to and consuming from a topic using the console is definitely convenient, but there are a number of other methods to use to produce data to a Kafka topic. For example, in practice, you'll probably be using the producer API from your application or even Kafka Connect. But for now, let's explore another manual method for writing data and check out the Confluent Cloud command line interface. Before we jump in, we need to set up the command line interface. At the bottom left-hand corner of your screen, you'll see the CLI and tools tab. From here, you'll find instructions on how to download and update the command line tools that we'll be using. Once that's ready, head back to your terminal and run confluent login -save to prompt you for your username and password. Use the same credentials you've used to create your Confluent Cloud account. Note that the save flag simply allows your credentials to be saved locally to a file for future use. Next, we need to select our environment and a cluster. First, we set the environment and then within that, the cluster. To set the environment, you'll want to know which environments are available to you. Run confluent environment list to see a list. For our purposes today, we'll use the default environment, so make note of its ID. With that environment ID in mind, we can now run the use environment command that you see here. With the environment set, we can now choose which Kafka cluster within it we'd like to use. We can see which Kafka clusters are available to us using confluent kafka cluster list. Note the cluster resource ID field of the results that are displayed. Enter it in the use Kafka cluster command as seen here. We've now told the CLI which environment and cluster to use. Before we can start producing it to the cluster, we need to provide an API key in order to connect to the cluster. Thankfully we can use the command line interface to do this as well. Using the cluster resource ID that you found earlier, you can run the create api-key command. You'll see an API key and secret printed out for you. Be sure to this securely somewhere. Next, we tell the CLI to use that key. Run the use API key command and enter the key and resource details. When that's done, we're ready to start putting some data into motion. With the command line interface setup, we can now start to interact with our Confluent Cloud cluster by consuming and producing messages. Let's first list out all of the topics available to us. From a terminal window, you can run confluent kafka topic list. You should see the poems topic. To consume messages from the beginning of the poems topic, run the topic consume command as shown here. The from-beginning flag tells the consumer to start reading messages from the earliest message in the topic. So you should see the four messages that we wrote earlier. Let's produce some more messages to the topic. Leave the terminal window with your consumer up and running and open a new terminal window. We can produce a key and value at once using the parse-key functionality. When we run the topic produce command with this flag, we can specify a colon-delimited string that the CLI will parse into both a key and value. Enter the following strings when prompted. Almost instantly after the messages are produced, you should start seeing them appear in the terminal window where your consumer is running. After doing so, return to the web interface on Confluent Cloud to see your newly produced messages. Remember to view messages in the web interface, you need to specify an offset for a given partition from which to consume. Since these are the first messages in the topic, set the offset to zero and check each partition to see where the messages were written. And that's it, if you follow this exercise all the way through, you've accomplished a number of pretty cool things. You signed up for Confluent Cloud and created your first Kafka cluster, created a topic, added messages to that topic using the web console, then you installed the CLI and created an API key, and finally used the CLI producer and consumer. With all of that out of the way, you're well on your way to building great things with Confluent Cloud. But don't forget to continue your Kafka journey and learn more about the great features that Kafka has to offer you by joining me in the following exercises, see you there.