Configuring Durability, Availability, and Ordering Guarantees

Hey, everyone. Welcome back. This is Jun Rao from Confluent. In this module, we're going to talk about how to configure some of the durability, availability and ordering guarantees within Kafka. (playful music) In some of the previous modules, we talked about how the control plane and the data plane work within a Kafka cluster. In this module, we're going to switch gears a little bit and talk about some of the guarantees. Kafka Replication Let's first talk about some of the durability and availability guarantees. Earlier, we talked about Kafka replication being a great way of providing durability and availability. So when you create a topic, you can specify the number of replicas you want to have. If you specify N number of replicas, then we can tolerate N - 1 failures. For mission critical applications, you can potentially configure even more replicas to tolerate a greater number of failures. Only the committed data will be exposed to the consumers. Now what about the producers? The producers actually have some choice as to when to receive the acknowledgement. Producer Modes The simplest form for a producer is to have this acks equals zero mode. This is a little bit like a fire-and-forget mode. So it's pretty good for latency because you don't need to wait for the response to be sent, but there's no strong guarantee of durability because the data can be lost before it even reaches the broker. The second mode the producer can have is acks equals one. In this mode, the producer will wait for the data to be received and added into the log of the leader replica before the response is sent to the producer. In this case, the latency will be longer because you have to wait for this round trip between the producer client and the leader broker. The durability story is much improved but there's still a little bit of chance when there's something like a leader election while a message is being produced, some of the records or events could be lost. The strongest mode we have is called acks equals all Axis Mode If you configure this, then the producer will only be acknowledged once the data is fully replicated to all the in-sync replicas. This is actually the strongest mode for durability guarantees. And it's important for mission-critical applications. In terms of latency, it's a little longer than the acks that are smaller. It's roughly two point times longer than the acks equals one mode. Because in addition to the round trip between the producer and the leader, you have to wait for another round trip for the data to be replicated from a leader to the follower potentially in parallel, if you have multiple followers. But you have to wait another round, a half round trip, for the follower to acknowledge the receipt of those events. So it's roughly 2.5 times then, the latency of the leader. Minimum insync replicas Now let's talk about some of the other guarantees. For some of the mission critical applications, they might may want to say, if I produce some records I only want to have the records acknowledged successfully if it's guaranteed that records will be inserted into at least a certain number of replicas. Otherwise, I'd rather choose that they not be taken. So this can be supported with a top-level configuration called minimum in-sync replicas. If you configure this, then when you append the data to this log, it will only successfully replicate the data and acknowledge it if at that point, there are enough replicas, more than this minimum insync replicas in the insync replica set. If there are fewer replicas, then this configuration in ISR will send a "not enough replicas" error back to the producer indicating that your data is rejected. So this is another way for mission critical applications to strengthen some of their durability guarantees. Ordering guarantees Now let's switch gears a little bit to talk about some of the ordering guarantees. In the earlier modules, we have talked about how in general, we are already enforcing strong ordering from the producer all the way to the consumers. For the producers, for all the messages sent from a single producer to a given partition, they will be stored in the commit log for that partition in the order they are sent and they will be delivered in the same order to the consumer in normal mode. But sometimes, the ordering guarantee can be a little tricky if there are failures. Let's consider this example. For example, in this case, the producer is trying to send three records m1, m2, m3 into the leader. Let's say halfway through, maybe only two of the records have been fully replicated and considered committed. But at that point, the leader is dead. Or the producer didn't get any acknowledgement for all those three messages. Then in order not to lose those events, what a producer needs to do is to retry sending those three messages to the new leader. So it will send m1, m2, and m3 again. In this case, since the new leader doesn't know this is actually a duplicate of the previous send, it will just forcefully take those records in this log. Now you can see, we have introduced some duplicates because m1 and m2 have been received twice. And we are also introducing some of the violation of the ordering guarantees because now, m1 appears to happen after m2 even though technically, it's sent before m2. So to solve this problem, we have introduced another feature called idempotent producer. This is actually pretty easy to enable. You just need to set this configuration in the producer. Set or enable idempotency to true. Once you have set this mode, the producer will try to avoid those duplicates and reordering even when there are failures. So let's see how it works. The way this works is when the producer is first started in this mode, the first thing it will do is to ask the broker for a producer ID. This is a unique ID intended for the lifetime of this producer. Then each time the producer tries to send some of the events to the leader, it will tag all those events with the producer ID as well as the corresponding sequence number associated with each record. The sequence number is uniquely used to identify each of the messages the producer will be sending. Once the leader receives those records, it'll append those records with the pid and the corresponding sequence number in this commit log. And this will be replicated to other replicas as well. Now, let's say the same thing happens. About halfway, you only have two messages, m1 and m2 that have been fully replicated. And then the old leader dies. In this case, a new leader will be elected. And when the producer is trying to resend the same three events to the new leader, it'll be sending those three events with the same pid and sequence numbers. And once the new leader receives them, it can compare those sequence numbers with the last sequence number it has seen with this producer ID. In this case, it will notice that message one and message two, based on sequence number, are duplicates because it has seen them before in its own log. So it will only take the last message, which is m3, in its log. As you can see, with idempotency introduced, now we can truly avoid the duplication as well as avoiding changing the ordering of those events. So this leads us to ordering guarantees. End to end ordering guarantee If you have idempotency enabled in the producer and you also have acks equals all set in the producer, in general, you are guaranteed this end-to-end ordering guarantee. In particular, if you have a single producer sending keyed messages, we guarantee all the events sent with the same key they will be landing in the same partition in exactly the same order they are sent and they will be delivered to the same consumer in the exact order they are sent. So this is actually an end-to-end key-ordering guarantee that we support. That's it for this module. Thanks for listening.