Introduction to Apache Kafka Partitions

Hey, Tim Berglund with Confluent, here to tell you how Kafka partitions topics. Now you probably know Kafka is a distributed system so it's designed to operate across a number of computers and look to the outside world to you writing code against it like it's just really one computer, right? And given that, if a topic were constrained to live entirely on one machine, one node in that cluster, that would be a bit of a bummer. It would place a pretty radical limit on the ability of Kafka to scale, right? A topic could never get bigger than the biggest node you could deploy. And maybe you don't wanna deploy big giant nodes. Maybe it's more economical for you to deploy kind of a medium instance size or you're running on prem and there's a certain kind of server that you have a lot of. Whatever it is, you don't want that to be a thing. And Kafka, in that case, it could still manage topics across many machines. But since no one topic could ever get too big or aspire to accommodate too many reads or writes, that would be just unfortunate. Well, happily Kafka doesn't leave us without options here. It gives us the ability to partition topics. Partitioning takes the single topic log, remember the topic is just a log and breaks it into multiple logs each of which can live on a separate node in the Kafka cluster. This way, the work of storing messages, writing new messages, reading existing messages, all that kinda stuff can be split among many nodes in the cluster. You actually get to act like a distributed system. Of course having broken a topic up into partitions, we need a way of deciding which messages to write to which partition, right? You've got somebody writing messages to this topic, now that topic is in many pieces. I need to know which piece a given message goes to. Now typically if a message has no key, remember an event, a message is a key value pair, if that key is null, it's empty, then the messages that you write will be distributed round robin among the topic's partitions. You just go from one to the next, you're always filling partitions evenly. And that's kinda the nice thing about that. If the message does have a key then we use that key to figure out which partition to put the message in. What usually happens is that we run that key through a hash function and then take the output of that hash function, mod the number of partitions, and the resulting number literally is just the partition number to write to. The cool thing about this, is that it allows Kafka to guarantee that messages having the same key always land in the same partition and therefore are always in order. So if your key is null and you're just round robining, you're distributing evenly to partitions and you're able to scale a lot, and that's all good. But you're never really gonna know what order things came in so if order doesn't matter all that much, it's just sort of a proximate vague time order, then that round robin approach works. But if you do need order, then you pick a key that preserves that order. And I'll say again, messages with the same key, always go in the same partition and therefore will always be in strict guaranteed order. An example, if you're producing events that are all associated with the same customer, then using the customer ID as the key is gonna guarantee that all of the events from a given customer will always arrive in order when we read them back out. This creates the possibility of course that a very active key will create a larger and more active partition if there's one super, super active customer. But that risk is small in practice and is manageable when it presents itself as long as you don't have some bizarre power law situation where there's one customer that has a billion times more events than the next customer, that's usually not the case in the real world. When you have things that are normally distributed, it works out pretty well and it's definitely worth that small risk in order to preserve ordering by key. So, that very simply, is partitioning in Kafka.