Hands On: Partitioning

Hands On: Partitioning

Up until this point in the course exercises, we have used a topic with the default six partitions. As you’ve learned through the course modules, though, partitions are useful in allowing you to break up your topic into manageable chunks that can be stored across multiple nodes in your cluster. Through this exercise, you’ll learn how to create topics with different partitions using the command line interface and see how varying topic partition counts affect the distribution of your data.

Again, we recommend following these exercises in order so that you have everything you need to complete the exercise. If you haven’t already set up your CLI to connect to your Confluent Cloud cluster, take a look at the earlier exercise to catch up.

1. From a terminal window, list your topics.

confluent kafka topic list

You should see only the poems topic.

2. Describe the topic to see more details into the topic and its configuration values.

confluent kafka topic describe poems

In particular, make note of the num.partitions value, which is 6.

3. Create two more topics with 1 and 4 partitions, respectively.

confluent kafka topic create --partitions 1 poems_1
confluent kafka topic create --partitions 4 poems_4

4. Produce data to the topics using the produce command and --parse-key flag.

confluent kafka topic produce poems_1 --parse-key

When prompted, enter the following strings:

	1:”All that is gold does not glitter”
	2:"Not all who wander are lost"
	3:"The old that is strong does not wither"
	4:"Deep roots are not harmed by the frost"
	5:"From the ashes a fire shall awaken"
	6:"A light from the shadows shall spring"
	7:"Renewed shall be blad that was broken"
	8:"The crownless again shall be king"

5. Repeat step 4 for the poems_4 topic.

6. From the Confluent Cloud Console, view the newly produced messages in both topics. Note that the poems_1 topic has all eight messages in its single partition…

   

…while the poems_4 topic has a slightly different distribution.

7. Using the Jump to offset field, explore the partitions of all three topics—poems, poems_1, and poems_4—to observe how the messages are distributed differently across these topics.

This should have given you a good idea of how partitions will affect the distribution of data across your topic. The next time you create a topic, think about how many partitions you’ll need!