Here are some of the questions that you may have about Apache Kafka and its surrounding ecosystem.
If you’ve got a question that isn’t answered here then please do ask the community.
A Kafka topic describes how messages are organized and stored. Topics are defined by developers and often model entities and event types. You can store more than one event type in a topic if appropriate for the implementation.
Kafka topics can broadly be thought of in the same way as tables in a relational database, which are used to model and store data. Some examples of Kafka topics would be:
Topics can be partitioned, and partitions are spread across the available Kafka brokers.
To list Kafka topics use the
kafka-topics command-line tool:
./bin/kafka-topics --bootstrap-server localhost:9092 --list
Using Confluent you can also view a list of topics with your web browser.
Many Kafka users have settled on between 12 and 24 partitions per topic, but there really is no single answer that works for every situation.
There are a few key principles that will help you in making this decision, but ultimately, performance testing with various numbers of partitions is the safest route:
You can read more in this blog post by Jun Rao (one of the original creators of Apache Kafka®).
You can create a Kafka topic with the
kafka-topics.sh command-line tool:
./bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 \ --topic my-topic --bootstrap-server localhost:9092
confluent kafka topic create <topic> [flags]
Another option is the Confluent Cloud Console, where you can simply click the
Create topic button on the
While there is no set limit to the number of topics that can exist in a Kafka cluster, currently Kafka can handle hundreds of thousands of topics, depending on the number of partitions in each.
With Kafka's new KRaft mode, that number will be in the millions.
You can delete a Kafka topic with the
./bin/kafka-topics.sh --delete --topic my-topic \ --bootstrap-server localhost:9092
confluent kafka topic delete my-topic [flags]
Another option is the web-based Confluent Cloud Console, where you can click on the topic on the
Topics page, then go to the
Configuration tab and click
To delete the contents of a Kafka topic, do the following:
Change the retention time on the topic:
./bin/kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name my_topic \ --add-config retention.ms=0
Wait for the broker log manager process to run.
If you inspect the broker logs, you'll see something like this:
INFO [Log partition=orders-0, dir=/tmp/kafka-logs] Found deletable segments with base offsets  due to Retention time 0ms breach (kafka.log.Log)
Restore the retention time on the topic to what it was previously, or remove it as shown here:
./bin/kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name orders \ --delete-config retention.ms
A few things to be aware of when clearing a topic:
This hands-on course will show you how to build event-driven applications with Spring Boot and Kafka Streams.