How to read from a specific offset and partition with the Kafka Console Consumer

This tutorial requires access to an Apache Kafka cluster, and the quickest way to get started free is on Confluent Cloud, which provides Kafka as a fully managed service.

Make a local directory anywhere you’d like for this project:

Next, create a directory for configuration data:

From the Confluent Cloud Console, navigate to your Kafka cluster and then select CLI and Tools in the lefthand navigation. Click the CLI Tools header and get the connection information customized to your cluster.

Create new credentials for your Kafka cluster and Schema Registry, writing in appropriate descriptions so that the keys are easy to find and delete later. The Confluent Cloud Console will show a configuration similar to below with your new credentials automatically populated (make sure Show API keys is checked). Copy and paste it into a configuration/ccloud.properties file on your machine.

This tutorial has some steps for Kafka topic management and/or reading from or writing to Kafka topics, for which you can use the Confluent Cloud Console or install the Confluent CLI. Instructions for installing Confluent CLI and configuring it to your Confluent Cloud environment is available from within the Confluent Cloud Console: navigate to your Kafka cluster, click on the CLI and tools link, and run through the steps in the Confluent CLI tab.

Your first step is to create a topic to produce to and consume from. This time you’ll add more than one partition so you can see how the keys end up on different partitions.

Use the Confluent CLI to create the topic:

To get started, lets produce some records to your new topic.

Since you’ve created a topic with more than one partition, you’ll send full key-value pairs so you’ll be able to see how different keys end up on different partitions. To send full key-value pairs you’ll specify the parse.key and key.separator options to the console producer command.

Let’s run the following command in the broker container to start a new console producer:

Then enter these records either one at time or copy-paste all of them into the terminal and hit enter:

After you’ve sent the records, you can close the producer with a CTRL+C command.

Next let’s open up a console consumer to read records sent to the topic in the previous step, but you’ll only read from the first partition. Kafka partitions are zero based so your two partitions are numbered 0, and 1 respectively.

Confluent CLI currently does not have the ability to specify reading from a particular partition, so for the next few steps you’ll be using the console consumer built into a Docker image. Notice that you are passing in the path to the configuration/ccloud.properties file that you created earlier.

Lets start a console consumer to read only records from the first partition, 0

After a few seconds you should see something like this (your output will vary depending on the hashing algorithm):

You’ll notice you sent 12 records, but only 6 went to the first partition. The reason for this is the way Kafka calculates the partition assignment for a given record. Kafka calculates the partition by taking the hash of the key modulo the number of partitions. So, even though you have 2 partitions, depending on what the key hash value is, you aren’t guaranteed an even distribution of records across partitions.

Go ahead and shut down the current consumer with a CTRL+C

In the previous step, you consumed records from the first partition of your topic. In this step you’ll consume the rest of your records from the second partition 1.

If you haven’t done so already, close the previous console consumer with a CTRL+C.

Then start a new console consumer to read only records from the second partition:

After a few seconds you should see something like this

As you’d expect, the remaining 6 records are on the second partition.

Go ahead and shut down the current consumer with a CTRL+C

So far you’ve learned how to consume records from a specific partition. When you specify the partition, you can optionally specify the offset to start consuming from. Specifying a specific offset can be helpful when debugging an issue, in that you can skip consuming records that you know aren’t a potential problem.

If you haven’t done so already, close the previous console consumer with a CTRL+C.

From the previous step you know there are 6 records in the second partition. In this step you’ll only consume records starting from offset 3, so you should only see the last 3 records on the screen. The changes in this command include removing the --from-beginning property and adding an --offset flag

Here’s the command to read records from the second partition starting at offset 6:

After a few seconds you should see something like this

As you can see, you’ve consumed records starting from offset 3 to the end of the log.

Go ahead and shut down the current consumer with a CTRL+C