Confluent Platform 6.1 | What’s New in This Release + Updates

Tim Berglund:

Confluent Platform 6.1 is out. Multi-Region Clusters get an important update with self-promoting observers, ksqlDB gets easier to use with the all-important create or replace statement, and you get improved metrics and management tools in Control Center. Check out the details in this special episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.

Tim Berglund:

Hi, I'm Tim Berglund here on a chilly winter's evening in Littleton, Colorado, to tell you all about Confluent Platform 6.1.

Tim Berglund:

What's new in 6.1? Well, as usual, we've got the features divided up into the kind of three categories of interest. There's productivity, if you're using it as a developer, building applications, there's being efficient at operating it, if you are an administrator or an operations person and there's architectural concerns. What are the things you're building? What are the kinds of systems that you can economically conceive of building using Confluent Platform? And I want to tell you about just a few features that we've got lined up that fall fairly neatly into those three categories.

Tim Berglund:

And before I dig into the details, let me kind of slice these a different way. If you think about downtime, like you don't want your system to go down, I think we can all agree on that, then failover, multi-region failover is the thing we're going to want to work well. So there's some improvements there. Also, with ksqlDB queries, which normally you think of as just a development tool, right? There's an important new feature that helps reduce application downtime in ksqlDB queries as well. When you think about simplifying operations, again, that failover kind of situation, that's definitely an operations-feature. And Control Center. What we affectionately call C3. We've got some new features in there and a new status API that'll help you get visibility into what's going on in the cluster.

Tim Berglund:

So let's dig in with a new Multi-Region Cluster feature. Now in a previous release of Confluent Platform, we introduced observer replicas. An observer replica, just to recap, if you're new here, is like an asynchronous replica, okay? When a producer is producing and let's say it has acts set to all, and replication factor three, and min ISR of two or something fairly standard, single region cluster kind of settings, conservative settings. Then there's one leader and two followers, and those two followers synchronously need to... Or at least one of those two followers, synchronously needs to reply before the producer can move forward. That's a synchronous operation and that's how producing and followers interact. You actually wait for them to write the replica.

Tim Berglund:

An observer is like a follower, but it's asynchronous. It never gets in the way of a produce going forward. So it's still an extra, in this case, an extra copy of a partition, on a broker, but it's never going to make a produce slowdown. It's a very, very important part of the machinery of multi-region Confluent Platform.

Tim Berglund:

So the way folks have been using this, it's been a little touchy in failover situations because you've had to manually promote an observer to regular replica status. So if you've got a synchronous replica that fails in one of your regions in a Multi-Region Cluster, you've got an observer that can take over for that, but you have to promote it manually. And there are some things you have to be careful with to avoid data loss when you're promoting an asynchronous observer. If you think about that for a moment, that makes sense. If you know how writes and reads work in Kafka. So it's been very, very important piece of machinery and a good feature, but a little difficult in some cases to handle these failovers.

Tim Berglund:

Well, it gets better now. Replicas are automatically promoted to proper follower status in Confluent Platform 6.1. So if you have that kind of failure scenario in a Multi-Region Cluster, it's easier for it to remain available with that automatic promotion following rules to make sure data doesn't get lost. And the previous way of dealing with this was of course called the Anna pattern named after the famous Anna McDonald and with automatic replica promotion, that whole thing gets easier. And it's easier to stay up with what I'll call a prudently provisioned Multi-Region Cluster. So very, very important new feature. You want to check it out.

Tim Berglund:

The bottom line is you can achieve higher availability for self-managed that is not Confluent Cloud multi-datacenter Kafka with less manual input. And that's the thing that you want. All right? If you're operating these, you don't want to be doing that manual stuff. The cluster just acts like a multi-region animal a little bit more natively.

Tim Berglund:

Now, you've got a ksqlDB query and it's running and you want to change it. Quick, what do you do? Well, you go and you drop some things and you find the query and drop it, and it's kind of a pain. You make a new one and it's a few steps to do. It doesn't feel especially fluid. Well, now in Confluent Platform 6.1, ksqlDB brings us the create or replace feature. You see that here, create or replace stream S1, et cetera, et cetera. That's one way of modifying or running query. So if it doesn't exist, it creates it. If it does exist, it tears down the old one and gives you the new one.

Tim Berglund:

Also you've got alter stream, which works pretty much just like alter table. Like you'd expect it to from ANSI SQL. And you can see an example there where we're adding a couple of columns to a stream. So you want to check out the docs for a full ultra stream and create a replace syntax. These are just a couple of things that make ksqlDB a little more productive to use if you've been using it in rigorous, real, actual development mode or in anger, as we say, not that it makes you angry, but for real, you're going to know these are going to be great features that will make your life a lot easier.

Tim Berglund:

Also, we get what we call multikey pull queries or really the in predicate. So as you see here, we can do a select from, it says table. That's a table that we've created in a previous query. This is a pull query where the key is in the set of those values. So another handy evolution of the syntax of ksqlDB and a feature that I think you'll come to rely on pretty quickly.

Tim Berglund:

Confluent Platform offers a Cluster Registry and Confluent Control Center now integrates with that Cluster Registry. So we could have a group of clusters. We can refer to clusters by a name, not a GUID. I mean, maybe you like typing GUIDs, if you do, I think that should be your right. You should be able to type big, long, extremely counterintuitive alphanumeric strings, but Control Center's not going to make you do that anymore. We can actually give these things nice names and associate certain kinds of properties with the set of clusters that are registered. Like if you've got some, say security settings or back settings that you want to apply to all your clusters you can accomplish that through the cluster registry interface. So that's just a thing you're going to want to check out. It just makes managing groups of clusters easier and kind of upgrades that to a slightly more first-class citizen of the control center universe.

Tim Berglund:

And also we've added Self-Balancing Cluster operations to the status API. So now you can create queries that are going to return results that have to do with self-balancing operations. So that's partitions moving around, brokers being created and destroyed, these kinds of things that are going to happen as cluster events during self-balancing, those have all now come into the status API. So that's an important thing that honestly, you will need to monitor if it's happening and you can now happily monitor it through the status API, which should be a great thing, because that really is a great, powerful API.

Tim Berglund:

If you're not using it, by the way, do check it out. Hit the docs as soon as you can. If monitoring is a concern of yours, you really want to know about that feature. This is just a small thing being added to it, but it's super important and I encourage you to check it out.

Tim Berglund:

So that's Confluent Platform 6.1 in a nutshell, here on a cold and dark evening. As usual, I encourage you to download it as soon as you can. Check it out and let us know what you're building.

Tim Berglund:

Hey, you know what you get for listening to the end? Some free Confluent Cloud. Use the promo code 60PDCAST, that's 6-0-P-D-C-A-S-T to get an additional $60 of free Confluent Cloud usage. Be sure to activate it by December 31st, 2021, and use it within 90 days after activation. And any unused promo value on the expiration date will be forfeit and there are limited number of codes available. So don't miss out.

Tim Berglund:

Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me @tlberglund on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you'd like to join. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through Apple Podcasts, be sure to leave us a review there. That helps other people discover us, which we think is a good thing. So thanks for your support and we'll see you next time.