The hidden cost of batch processing

Batch updates can silently put a business out of sync with reality. Imagine a retailer whose $1,000 smartphone sells its last unit at 9:01 AM. Inventory is synced to the sales platform by an hourly batch update, a common practice even though companies like Walmart have moved from batch inventory to real-time event streaming. So the website keeps accepting orders for a phone that no longer exists, each with a next-day delivery promise, until the batch job runs at 10 AM. Rerouting those orders turns next-day delivery into three-day delivery, and the business's defining feature becomes a liability.

Klang's Conjecture, coined by Viktor Klang, says: "If you cannot solve a problem without programming; you cannot solve a problem with programming." In a physical store, every action is an event that happens in real time, and a batch process puts an artificial constraint on those events that only exists in the software. Stream processing removes it: the sale publishes a "Product Sold" event, inventory updates its counts, and a "Product Sold Out" event stops the website selling the phone, shrinking the delay from 59 minutes to moments. This shifts data engineering upstream ("shift-left analytics"), with cleaning and transformation happening in-stream. The benefits are lower latency, better reliability, and less duplication, as data engineers become data stream engineers.

Topics:

• How data silently falls out of sync with reality
• Business impact of a 59-minute data delay
• Klang's Conjecture and batch processing as an artificial constraint on real-time events
• Stream processing and "shift-left" analytics
• Modeling real events: "Product Sold" and "Product Sold Out"

Resources

• Walmart's Real-Time Inventory System Powered by Apache Kafka - Walmart engineering's description of its event-streaming-based, real-time inventory system
• Klang's Conjecture - Viktor Klang on X - Original source for Klang's Conjecture