Why one data pipeline can't serve everyone

Serving is the last mile of the data engineering lifecycle, and the danger is designing for how we think people will use our data rather than how they actually do. We bucket consumers into three use cases and ask two questions of each: what is the access interface, and what is the hard part? Analysts and auditors use BI tools over SQL, where the hard part is scalability. Data scientists need training history and millisecond inference lookups, where the hard part is consistency across stores. Operational applications need push-based access, where the hard part is latency.

No single physical output serves all three, so the serving layer multiplexes one logical dataset into several physical shapes. Every fix makes the same move: shift compute left, upstream, and pay at write time instead of query time. Because a shared pipeline becomes a multi-tenant platform, per-tenant observability lets consumers answer their own questions about lag, stale data, schema breaks, and cost.

Topics:

• Why we are not the consumers of our data
• Three consumer use cases: BI and analytics, machine learning, and operational applications
• Access interface and the hard part for each consumer
• Shifting compute left: pay at write time instead of query time
• Serving as a multi-tenant platform

Resources

• Other Databases (JDBC) - Tableau - Tableau's documentation for connecting to databases through a JDBC driver
• What is ClickHouse? - ClickHouse documentation describing it as a column-oriented SQL database for online analytical processing (OLAP)
• What is a Feature Store? - Databricks - Explains training-serving skew, offline and online feature serving, and reusing one feature definition across environments
• Data Contracts for Schema Registry on Confluent Platform - Data contracts as formal agreements between producers and consumers, and migration options for breaking schema changes