Under the Covers of ksqlDB

Under the Covers of ksqlDB

You have learned about ksqlDB's syntax and functionality, but it can be beneficial to get a sense of its internals.

When multiple ksqlDB servers are organized as a cluster, partitions are used as the basis for scaling the workload. Just like Apache Kafka consumers or Kafka Streams, ksqlDB can parallelize throughput by allocating work on partitions to different nodes. In Confluent Cloud, both your ksqlDB applications and Kafka clusters are fully managed. But you can also run ksqlDB in self-managed mode to run a custom user-defined function, or for other reasons. In this case, you can still take advantage of Confluent Cloud to manage the Kafka cluster used by your ksqlDB applications.

ksqlDB consumes and produces from Kafka topics, using SQL declarations to build and execute Kafka Streams topologies.

RocksDB is used internally to store state for tables and aggregations, with Kafka topics providing a changelog for RocksDB.

Multiple ksqlDB servers can be grouped together to form a cluster, for parallel processing and resilience.

Organizations will usually deploy multiple clusters of ksqlDB servers, forming a topology much more akin to application deployments than to a behemoth RDBMS.

Read more in the ksqlDB documentation and within the Confluent Developer course Inside ksqlDB. You can also find some deep-dives on ksqlDB functionality on the Confluent blog, including:

• How Real-Time Stream Processing Safely Scales with ksqlDB
• Bounding ksqlDB Memory Usage
• State Store in Recovery in ksqlDB
• Consistent Metastore Recovery for ksqlDB Using Apache Kafka Transactions