Building a Streamhouse

From the age of BI to the age of AI

The course is framed around a change in who consumes data. In the age of BI, the consumer was a person asking questions about the past, on their own schedule. Warehouses and lakehouses are excellent at that job, and they aren't going anywhere.

Today, more systems are being built as software that reacts to events: an app showing a customer where their parcel is, a rule deciding whether to block a transaction, a service deciding whether a truck needs maintenance. In this course, "AI" means the whole class of automated consumers, not just LLMs. They query continuously, act immediately, and have no human in the loop to notice that an answer is six hours old. The failure mode shifts from "a stale dashboard" to "a confident, wrong action."

Keeping the current state of the business

Every business runs on data that changes constantly, but most systems only ever see a stale copy of it: a nightly batch load, a cache that's minutes behind, a dashboard built from yesterday's extract. For a person reading a report, that's usually fine. For software that acts on the answer immediately, thousands of times a day, it isn't.

This course is about building systems with an architecture that fills that gap, the Streamhouse. You'll start from a concrete problem in each lesson, add the piece that solves it, and then look at what that piece still leaves unsolved. By the end, you'll have built the whole pattern: first from open source parts running in Docker, and then on Confluent Cloud.

You'll learn about:

• Data Capture: Capturing a change once, at the moment it happens, instead of re-extracting it on a schedule.
• Event Streams: The durable, ordered record of what happened, and why topics work as contracts between teams.
• Stream Processing: Maintaining an always-current result incrementally, instead of recomputing it in batch.
• Governance at the Source: Schemas, access control, and lineage applied to the stream itself, not to a downstream copy.
• Serving: Why a raw log can't answer "what's the current value for customer 7521?", and how open tables meet this need.
• Real-Time Detection: Catching problems right away, using state that's always current.
• Resilience: Recovering by replaying the durable log, rather than restoring from a backup.
• Agents as Consumers: Grounding an agent in live, governed state through a plain serving interface, and routing its decisions back to the point of action.
• Incremental Adoption: A maturity curve that shows where to start and what each step delivers.

What a streamhouse gives you

A streamhouse doesn't replace your warehouse or lakehouse. It sits alongside them, and the course is explicit about where each one belongs. What it adds:

• One current, governed, shared state: The business's current state is maintained incrementally, governed from the edge, and read by applications, analytics, and agents, with no per-consumer copy.
• Always-on: When software depends on that state continuously, the layer that maintains it has to meet an operational bar, not a reporting one.
• Decoupled and open: Built on open standards (Kafka, Flink, Iceberg-style open tables), so teams evolve independently and any engine can read the data.

Intended audience

• Developers who want to understand, concretely, what a Streamhouse is, when it's useful, what problems it solves, and where challenges remain.

Prerequisites

Required knowledge:

• Comfortable reading and writing basic SQL queries (joins, aggregations).
• Conceptual familiarity with replication, partitioning, and eventual vs. strong consistency. You don't need to have built a distributed system, just understand the concepts.
• Able to run and modify a docker-compose.yml file and read container logs.
• Comfortable with JSON payloads and simple HTTP requests.

No prior experience with Apache Kafka, Apache Flink, Debezium, or Apache Iceberg is assumed. Each one is introduced at the point it's needed.

Going deeper

This course stays at the level of the pattern. It covers enough Kafka, Flink, and Iceberg to make the architecture feel real, but it isn't a substitute for learning any one of them in depth. When you're ready to go further:

• Apache Kafka 101 for topics, partitions, and Kafka Connect.
• Apache Flink 101 for stateful stream processing and Flink SQL.
• Apache Iceberg for open table internals, schema evolution, and catalogs.
• Schema Registry 101 for data integrity, governance, and schema evolution.

---

Do you have questions or comments? Join us in the #developer-confluent-io community Slack channel to engage in discussions with the creators of this content.