Consumer Groups

What is a consumer group in Apache Kafka?

Consumer groups allow Kafka consumers to work together and process events from a topic in parallel. Consumers are assigned a subset of partitions from a topic or set of topics and can parallelize the processing of those events. (Alternatively, consumers can work individually to consume the same stream of events and process those events in different ways.)

What happens when consumers are added or removed from a consumer group?

When consumers are added or removed from a consumer group—either because of a changing workload requirement, a rolling machine, or a consumer failure—the Kafka brokers will redistribute the topic-partitions across the new or remaining active members within the group.

How do I use a consumer group?

To start using consumer groups, just give multiple consumers the same group.id. The group.id can use any string you like, and all the consumers with the same string will be in the same group.

For example, you might have 5 consumers all running with group.id=invoice-emails, and they will work-share; at the same time, a single consumer can read the same events from the same topic with group.id=sales-analytics and it will work completely independely.