Get Started Free
‹ Back to courses
course: ksqlDB 101

Under the Covers of ksqlDB

2 min
Allison

Allison Walther

Integration Architect (Presenter)

Robin Moffatt

Robin Moffatt

Principal Developer Advocate (Author)

Under the Covers of ksqlDB

You have learned about ksqlDB's syntax and functionality, but it can be beneficial to get a sense of its internals.

When multiple ksqlDB servers are organized as a cluster, partitions are used as the basis for scaling the workload. Just like Apache Kafka consumers or Kafka Streams, ksqlDB can parallelize throughput by allocating work on partitions to different nodes. In Confluent Cloud, both your ksqlDB applications and Kafka clusters are fully managed. But you can also run ksqlDB in self-managed mode to run a custom user-defined function, or for other reasons. In this case, you can still take advantage of Confluent Cloud to manage the Kafka cluster used by your ksqlDB applications.

ksqldb-kafka-db

ksqlDB consumes and produces from Kafka topics, using SQL declarations to build and execute Kafka Streams topologies.

RocksDB is used internally to store state for tables and aggregations, with Kafka topics providing a changelog for RocksDB.

Multiple ksqlDB servers can be grouped together to form a cluster, for parallel processing and resilience.

kafka-cluster

Organizations will usually deploy multiple clusters of ksqlDB servers, forming a topology much more akin to application deployments than to a behemoth RDBMS.

Read more in the ksqlDB documentation and within the Confluent Developer course Inside ksqlDB. You can also find some deep-dives on ksqlDB functionality on the Confluent blog, including:

Use the promo code KSQLDB101 to get $25 of free Confluent Cloud usage

Get Started

Get started with Confluent Cloud

Try free

Total 28

  1. Introduction to ksqlDB
    2 min
  2. Interacting with ksqlDB
    2 min
  3. Hands On: Interacting with ksqlDB
    5 min
  4. Creating, Exporting, and Importing Data
    2 min
  5. Filtering with ksqlDB
    1 min
  6. Hands On: Filtering with ksqlDB
    1 min
  7. Lookups and Joins with ksqlDB
    2 min
  8. Hands On: Lookups and Joins with ksqlDB
    2 min
  9. Transforming Data with ksqlDB
    2 min
  10. Hands On: Transforming Data with ksqlDB
    2 min
  11. Flatten Nested Records with ksqlDB
    1 min
  12. Hands On: Flatten Nested Records with ksqlDB
    2 min
  13. Converting Data Formats with ksqlDB
    2 min
  14. Hands On: Converting Data Formats with ksqlDB
    2 min
  15. Merging Two Streams with ksqlDB
    1 min
  16. Hands On: Merging Two Streams with ksqlDB
    2 min
  17. Splitting Two Streams with ksqlDB
    2 min
  18. Hands On: Splitting Two Streams with ksqlDB
    2 min
  19. Streams and Tables
    2 min
  20. Stateful Aggregations (Materialized Views)
    2 min
  21. Hands On: Stateful Aggregations (Materialized Views)
    2 min
  22. Push Queries and Pull Queries
    2 min
  23. Hands On: Push Queries and Pull Queries
    2 min
  24. Under the Covers of ksqlDB
    2 min
  25. ksqlDB Code Lifecycle
    2 min
  26. Apply Lambda Functions to Arrays and Maps
    2 min
  27. Hands On: Apply Lambda Functions to Arrays and Maps
    2 min
  28. Monitoring ksqlDB
    1 min

Related Resources

Presentation

Inside ksqlDB

Docs

How It Works

Docs

Core Concepts

Docs

Materialized Views

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.

Under the Covers of ksqlDB

Hi, I'm Allison Walther with Confluent. Let's take just a peak under the covers of ksqlDB to understand a bit about how it runs. We're not going to go into a deep dive here. There's lots of good deep-dive material about ksqlDB over on the Confluent Blog. ksqlDB runs separately from the Kafka brokers, which by the way, all of your Kafka applications should be doing. Kafka brokers are just for Kafka brokers. ksqlDB is built on top of Kafka streams and like Kafka streams, it uses RocksDB as its state store. Just like Kafka, ksqlDB is a distributed system that uses partitions as its basis for scaling its workload. We can have a single node or we can scale out the nodes and they form a cluster. Just like Kafka consumers, or Kafka streams, ksqlDB can parallelize throughput by allocating work on partitions to different nodes. But don't think of it as a single huge database cluster. ksqlDB queries are continuously running in processing events, their applications. So when it comes to deploying ksqlDB, think in terms of deploying applications. In Confluent Cloud, both your Kafka clusters and ksqlDB applications are fully managed. You only need to think about your queries and topics. But sometimes, you may need to run ksqlDB in self-managed mode. For example, in order to run a custom user-defined function. In this case, you can still take advantage of Confluent Cloud to manage the Kafka cluster that your ksqlDB applications are hitting. Again, allowing you to put more focus on solving business problems. If you wanna know more about ksqlDB internals, we have a course called Advanced ksqlDB. Go check it out.