ksqlDB's Architecture

Hi, I'm Michael Drogalis of Confluent. And in this module, we're gonna be looking at ksqlDB's Architecture. The best way to understand ksqlDB's architecture is to look at what it replaces. Here I have a typical architecture for working with data in motion. There are some databases and applications on the left, they're creating events. There are connectors that are harvesting those events and moving them into Kafka for storage and their stream processing for continuously transforming and aggregating that event data as it arrives. And finally, there's a set of sync connectors that move that aggregated data into a database that my applications can run queries over it. This works, but it's rather complicated. There are a lot of moving parts. It's hard to scale, secure, monitor, debug, and operate all of these components is one. What if this approach could be replaced by a simpler architecture? In fact, what if it looked something like Postgres? Now this might sound like a strange question. What does Postgres have anything to do with stream processing? Quite a lot, actually. Relational databases and in particular SQL have been an incredibly effective abstraction for creating simple architectures, SQL hides all of the underlying complexity of query execution, indexing, concurrency control, crash recovery, replication, and so on. Stream processing it turns out needs to implement many of the same pieces of technology to be a complete solution. That is why ksqlDB builds on this timeless abstraction and tucks away the complexity we just looked at behind the interface of a database. Inside you have primitives for stream processing, but you also have primitives for connectors, which make it easy to transport data to and from the outside world. And lastly, you have support for materialized views, which create data sets that can be queried directly by your application, just like a database table, because ksqlDB relies on Kafka to maintain all durable state, you get a simple two tier architecture, compute and storage. KsqlDB handles compute and Kafka handles storage. Both can be elastically scaled independently from one another. The way you interact with this architecture is with a simple client server deployment, just like a relational database. You have a CLI that you issue commands from, and that CLI sends those commands to one or more servers that can cluster together. Those servers execute those commands and communicate results back to the clients as needed. And beyond the CLI you can also use a full featured UI, a REST API or a programming language client like Java. The easiest place to try out ksqlDB is Confluent Cloud. As we work through the lessons in this course, you should sign up and do just that. Here's a promo code to get you started for free. To get started, go to the URL on the screen and click the Try Free button. Then enter your name, email and password. This email and password will be used to log into Confluent Cloud later. So be sure to remember it, click the try free button and watch your inbox for a confirmation email to continue. The link in your confirmation email will lead you to the next step where you can choose between a basic standard or dedicated cluster. The associated costs are listed, but the start-up amount freely provided to you will more than cover everything you need for this course. Click Begin Configuration to choose your preferred cloud provider, region and availability zone. Cost will vary with these choices, but they're clearly shown on the bottom of the screen. Continue to set up billing information, here you'll see that you receive $200 of free usage each month for your first three months. Also by entering the promo code KSQLDB101 you receive an additional $101 to give you plenty of room to try out the things that we'll be talking about. Click Review, to get one last look at the choices you've made, then launch your new cluster. While your cluster is provisioning. Join me in the next module in this course.