Introduction to Confluent's Offerings

This is the introduction to Apache Kafka about Confluent's offerings. This course has been about Apache Kafka. It's very important that you understand it by itself and that you understand ways to run it in the cloud without having to operate infrastructure on your own. That's what Confluent does.

So this is our view as of the time of this recording of what a data streaming platform looks like. Stream, connect, govern, process. These are all the things that you're going to have to do with your data. It's not just about Kafka. Of course, let's start with Kafka because that is the substrate, the foundation of everything we're doing.

Very broadly, the spectrum of Confluent's Kafka offerings go from Confluent Cloud, that's a fully managed cloud service. On the other end of the spectrum, over on the right, you've got Confluent Platform. That is Confluent's product for you to manage on your own infrastructure, whether it's physical machines on premises or instances in the cloud, whatever that is. If you, for some reason, need to run that yourself, that's Confluent Platform.

And in the middle, Confluent WarpStream, a relatively new edition as of the time of this recording, that offers a really compelling different architecture to run Kafka where all of the data is stored in S3 or the cloud blob store of your choice. And this gives you a really, really compelling bring your own cloud kind of solution where all of the data is yours. Effectively, the components that act like brokers, they're all in your infrastructure. Confluent never has any access to any of your data, can't. It's not just that we don't. We like don't have the ability to see your data. Super cool stuff.

That's optimized for latency-insensitive use cases since all the data is in S3. There is a trade-off with latency, but again, really, really compelling stuff. You should take a look at it. So Confluent Cloud, Confluent Platform, Confluent WarpStream, these are your options.

And inside Confluent Cloud, even there, there's a lot to do. Okay, you can create what's called a basic cluster. Super loose uptime guarantees. There are a lot of enterprise security features and things that aren't there, but it also scales to zero. If you don't use it, it doesn't cost you anything at all. That's great for testing, simple projects like that. Maybe things you're just trying to learn about. That's a great way to go.

Standard clusters, those are production-ready, good for most use cases. Then you've got enterprise clusters. Those things really scale. Any throughput, any latency, low latency, that is. These are fast clusters. All of the enterprise security features are there.

What if maybe you don't care so much about latency? Kind of like WarpStream, right? But inside the Confluent Cloud box here, we've got freight clusters. Again, the data is stored in S3 or cloud blob store of your choice. And you've got now a somewhat higher latency, but radically lower cost way of doing Kafka. Really cool stuff going on there with the engine under the covers. We've got some other videos that you'll probably be able to find fairly easily that talk about how freight clusters work, how the Kafka engine works under the covers there. They're beyond our scope right now, but it's super cool stuff to look into.

Then you've also got dedicated clusters over on the extreme high end. That's where you know you just have radically high throughput and you want dedicated hardware in the cloud, not shared infrastructure, but you're willing to say, these are the machines that I want to stand up and to be my cluster and only my cluster. That's kind of for some extreme scale use cases on that. But whatever you need on the whole spectrum, there's a kind of Kafka for you.

But the story doesn't end with Kafka. What about Connect? What about Govern? What about Process? We've talked about Kafka Connect all this time. I've talked about the schema registry. We've talked about Flink. These are the things that exist in this top layer. There's a pretty substantial ecosystem of prebuilt connectors. There are 80 of them available in fully managed form in Confluent Cloud. If you need to write a connector for some reason, that might be fun, that might not be fun, I don't know. But if you need to do it, we can run those with our Bring Your Own Connector support. Lots of Connect options there.

Under stream governance, it's not just about us running a schema registry instance for you. We've also got stream quality, a catalog on top of schema registry and data lineage support in the stream lineage features. There's a lot of interesting governance technology in the platform too.

Finally, processing. This is simple serverless stream processing with Apache Flink. You can type SQL into a box in Confluent Cloud and create a stream processing job just like that. There's also support for the Table API, applications written in Java or Python, if you prefer that. But you get unified monitoring security and governance for Kafka plus Flink without you having to stand up a Flink cluster of your own.

And of course, this runs in whatever cloud you want. Sometimes there might be a new feature that rolls out on one cloud before the others. But in general, Confluent Cloud runs on AWS, Azure and Google Cloud, whatever cloud you're in, we're in.

So this platform is a rapidly evolving thing. It's a moving target. So you should always go to confluent.io for up-to-date details about what's going on in Confluent Platform, what's going on in Confluent Cloud, WarpStream, all those things. And make sure you stay close to developer.confluent.io for more developer-focused educational material like this course.

I hope this has been a good and accessible and thorough enough introduction to Kafka and its related ecosystem. And as always, I really look forward to hearing what you build with it.