Confluent Platform 6.2 | What’s New in This Release + Updates

Tim Berglund:

Hi, I'm Tim Berglund with Confluent. Here, we're not in any exotic outdoor location this time, but in my studio at the house of Berglund, to tell you all about Confluent Platform 6.2. To build really robust systems that harness data in motion, to deliver rich frontend customer experiences with scalable performance real-time backend operations, you need specialized infrastructure different from what we've had in the past. That's Confluent Platform. Here data is not static and at rest, but continuously moving and continuously being processed and analyzed in real time. That's the key point. The notion is the data is moving, it's not just sitting there. For a modern software-defined business, a platform for data in motion is crucial to connecting every part of that businesses, large and non-trivial IT infrastructure together. It enables the business to process, react, and respond to their streams of event data in real time. And when I say the business, you and I both know that we mean software you're writing. What happens if we don't do this? Well, bad things. When mission-critical processes and applications fail, the business notices and not in a good way. Mitigating the risk of business disruption is critical for you and the systems you build to be able to compete and maybe even innovate and maybe win in the event-first digital world.

Tim Berglund:

With the release of Confluent Platform 6.2, we're introducing a feature called Health+. This provides the tools and visibility needed to ensure the health of your environment and minimize business disruption, which as we discussed we don't like. It gives us a few new things, intelligent alerts, cloud-based monitoring and a streamlined support experience. Let me tell you about these things, first up intelligent alerts. So these allow you to get customizable rule-based alerts to identify problems before they become critical issues that lead to things we don't like downtime and business disruption. We've managed over 5,000 clusters as of the time of this recording in Confluent Cloud. So we've had the ability to build up some interesting algorithms around alerting. Now that functionality is available to you, even if you're not a Confluent Cloud user.

Tim Berglund:

Next, a cloud-based monitoring dashboard. Again, extracted from our development of Confluent Cloud, monitoring dashboards allow you to look at all the critical health metrics of your clusters in a single dashboard that we serve to you. You don't manage extra monitoring infrastructure of your own because life is short and it's not just a dashboard, it's Confluent-backed insights and recommendations. That's in the same vein as intelligent alerts there. And of course you can integrate the monitoring data into your existing tools like Prometheus and Grafana and all that.

Tim Berglund:

Third is accelerated Confluent Support. Because the confluent support team has secure access to your cluster metadata. That's not data in topics, mind you, but metadata without you having to manually input information about your environment via support tickets in a moment when you're maybe possibly slightly, a little bit stressed, and not at your best, this ensures a much smoother support experience and hopefully reduced time to issue resolution. Like imagine if you could use a computer to write your Zendesk tickets for you. I mean, how great is that? Now digging into intelligent alerts just a little bit more. There are 10 total alerts right now as of this release, including validation of disk usage, unused topics, offline partitions and other things. Now, you know full well, product managers will not be able to contain themselves, they will add more alerts to this collection in the future and that is a very good thing. So count on this list to grow. You can also customize the types of notifications you receive, right now you're able to choose Slack, email, and webhook.

Tim Berglund:

Health+ also provides real-time and historical monitoring data aggregated and made visible to you for easy troubleshooting and trend analysis. You can view broker throughput, topic throughput, replicated partitions, disc usage, network handler pool usage, request handler pool usage and even more than that. Health+ services all this stuff clearly, and in an organized way, enabling you to dig into problems and analyze usage without having to run so much as a single local monitoring server. And remember the auto support feature I mentioned. Well, Health+ provides Confluent support engineers with critical context about your environment automatically. Again, why not use computers for these things? By enabling access to configuration and cluster metadata through Health+, we have a real-time view of the performance of your console platform deployment without having to access any of your clusters' data. And let me say that again, because it's important. The data that we can see only includes nonsensitive metadata, not payload or topic data. Sharing this context can help resolve Confluent Platform support tickets significantly faster and it offloads many of the manual steps, which nobody really enjoys required to file a support ticket.

Tim Berglund:

But Health+ isn't all there is in CP 6.2. Let me tell you about the rest. We're introducing new enhancements to cluster linking which is still in preview state for Confluent Platform. We're introducing new failover command, which makes it even easier to do an HADR failover using cluster linking. If you create a disaster recovery cluster in another region and have synced data using cluster linking, you can call failover on a per topic basis to keep up and running if your main cluster goes down. So, better disaster recovery and better cluster availability. In the ksqlDB department, we're enabling any serialization format to be used for keys in ksqlDB. Previously, you didn't have as many options typically of keys with just primitive types and if you want a big Avro object for your key, a lot of people do now, you actually can. 0.17 is the version of ksqlDB. In Confluent Platform 6.2, it's also got support for query migrations, very important, Lambda functions, array, instruct, key types if you really like those complex and compound key types and more. You wanna check the release notes on ksqlDB.io on that website for full coverage of the release.

Tim Berglund:

Following the longstanding tradition of every Confluent Platform release, 6.2 is built on the most recent version of Kafka, in this case, 2.8. 2.8 does a lot much to tell you about two things. Number one, it's the initial merge of KIP-500 which replaces ZooKeeper with a self-managed quorum. That's Kafka's own inter broker quorum protocol, but it's not possible to start a cluster without ZooKeeper and it goes through some basic product and consumer use cases. It's a huge step for Kafka. Another one, KIP-700. This is an enhancement to an admin API. It's the add described cluster API. The Kafka AdminClient has historically used the broker's metadata API to get information about the cluster. However, that is primarily focused on supporting the consumer and producer clients and they follow different patterns than the AdminClient. KIP-700 decouples the AdminClient from the metadata API by adding a new API to directly query brokers for information about the cluster. This change enables the addition of new admin features in the future without disruption to the producer and consumer and is also related broadly to the removal of ZooKeeper.

Tim Berglund:

And there's so much more than that. We've got a release video as always, a blog post, and the Apache Kafka release notes are always where you wanna go for full details. And that does it for Confluent Platform 6.2. Check it out today.