Scaling and Fault Tolerance

Hi, I'm Michael Drogalis with Confluent. And in this module, we're gonna be looking at how scaling and fault tolerance work with ksqlDB. What we've looked at so far is really just having one server and how it's able to recover when a new node takes place. But what happens when you have more than one server in your cluster? How does compute scale? How does fault tolerance work? Let's look at all of that now. We'll start by examining how scaling works in ksqlDB in the most basic sense. Now I've omitted the query here. It can be any stateless query that you like. But when you have one server, it will be assigned all the partitions for the stream or tables that it's reading from. In this case, it's reading from eight different partitions. This server is doing all of the work. It's processing all of the data that's available. In earlier modules we learned how ksqlDB inherits all of the properties of the Kafka consumer protocol. That means that when you add servers, it also inherits the properties of the consumer group protocol. Here, I add another server to the cluster, depicted with labels A and B. They're running the same persistent query, but now it's scaled out. The work for all the partitions is shared across the two nodes. The work rebalances for you automatically. So you get some number of partitions assigned to server A and some to server B. And this will be evenly distributed. Now, ksqlDB is processing the data twice as fast. The same will hold true if you add eight servers. It's processing the data with the maximum amount of parallelism. Notice how much faster this animation is playing than the others. It's not sped up. It's that it's actually running with 8X parallelism so it can process the data faster. KsqlDB will rebalance and reassign the right processing work to the right servers at the right time. When you scale up, this happens dynamically, safely and automatically. What we just saw was how scaling works in a stateless context. But what happens when you're doing a stateful operation like materializing a table. Here, it does the same thing, but the state needs to be sharded across each of the servers. Remember how we talked about data locality? This is a prime example of how a server needs to have the data it's going to process locally available. In this case, the state is sharded by key. So if I had servers A, B, C and D, and I have sensors one through eight, it will reshuffle the data to make sure that the same rows for the same key always go to the same partition and therefore to the same server in the cluster. Here again, the backing data for each piece of state is written out to a change log. So if a node in the cluster is lost, a new node can step in and read the change log to recover. So imagine we lost server B, the remaining partitions might be reassigned to servers A and C. The change logs would be used to repopulate those tables. It combines its usage of the client protocol, the consumer group protocol and the change log designed to scale state in a fault-tolerant way. That's how scalability and fault tolerance work in ksqlDB.