Lambda vs Kappa vs Data Streaming Platform — which one wins?

Every data pipeline, no matter how complex, follows the same simple pattern: sources produce data, transformations apply business logic, and sinks store the results. This Source-Transform-Sink model is universal — it doesn't care whether data arrives in nightly batches or continuous streams. But the architecture we choose to implement that model has enormous consequences for our teams, our reliability, and our sanity.

Most organizations arrive at their first streaming architecture by accident. A batch pipeline is humming along, and then someone in leadership asks for a real-time dashboard. Rather than redesign from scratch, the team bolts a streaming layer alongside the existing batch system. This is the Lambda Architecture: a batch layer for historical accuracy and a speed layer for real-time data, stitched together by a serving layer. It works — and it powers massive companies — but it comes with a serious operational tax. Two pipelines means two codebases, often two teams, and business logic implemented in two different engines. When the numbers inevitably diverge, debugging means tracing through both systems to find where the data split. "Eventual consistency" in Lambda often just means "inconsistent until someone manually fixes it."

The Kappa Architecture starts from a beautifully simple insight: batch is just a special case of streaming. An unbounded stream has a start but no end; a bounded stream has both — and that's just a batch. If the processing engine can handle both, there's no need for two separate pipelines. Kappa treats everything as a stream, uses one engine like Apache Flink® for all transformations, and keeps all business logic in one place. One codebase, one tech stack, one team, one place to debug. But teams often push back with practical concerns: storage costs for keeping data in the broker, analyst access to streaming data, and compatibility with existing query tools.

The Data Streaming Platform addresses those concerns head-on. The first move is to "Shift Left" — push transformations, governance, and data quality enforcement upstream into the stream itself so the work happens once rather than being duplicated by every consumer. The second is to sink the stream into an open table format like Apache Iceberg®, making the same data simultaneously available as a real-time stream for Flink and downstream microservices and as a high-performance table for Snowflake, Spark, or Trino. Everyone gets what they need from the same data, without separate pipelines for each consumer.

Topics:

• Dataflow as a universal mental model: Source, Transform, Sink
• Lambda Architecture: bolting streaming onto batch systems
• Operational cost of Lambda: dual codebases, dual teams, consistency headaches
• Kappa Architecture: treating batch as a special case of streaming
• Bounded vs. unbounded streams
• Common objections to Kappa and how to address them
• Shift Left: moving transformations and governance upstream
• Open table formats (Apache Iceberg) as a bridge between streaming and batch query engines
• The Data Streaming Platform: unifying real-time and analytical access

Resources

• Questioning the Lambda Architecture - Jay Kreps' 2014 article that critiques the Lambda Architecture's dual codebases and proposes the stream-only alternative now known as the Kappa Architecture
• Shift Left: Unifying Operations and Analytics With Data Products - Confluent's ebook on moving data processing and governance upstream to eliminate duplicate pipelines and enforce data quality at the source
• Apache Flink Architecture - Official documentation on Flink's unified batch and streaming processing model