Stop waiting for your data: how to take control

Ingestion is where the data engineer takes control of data as it moves into the pipeline. Taking control comes down to four dimensions: quality, format, timing, and access. Traditional ingestion methods, such as custom connectors, query-based change data capture (CDC), and batch or cron jobs, are often one-off and batch-oriented, which breaks the real-time flow of data.

Streaming alternatives keep the data flowing: Kafka Connect for declaratively streaming data from a wide range of back-end systems, event-driven CDC with tools like Debezium, and streaming platforms like Apache Kafka®. What matters isn't when data arrives, but what you can do with it once it does. With scheduled and batch patterns you wait for the data, then process it. With event-driven and streaming patterns you can process it as life happens, so build your pipeline for streaming even when the source is batch.

Topics:

• The core problem: data exists, but not where you need it when you need it
• The four dimensions of ingestion control: quality, format, timing, and access
• Traditional (batch-oriented) ingestion methods
• Streaming alternatives: Kafka Connect, event-driven CDC, and Apache Kafka
• Scheduled and batch patterns vs. event-driven and streaming patterns

Resources

• Kafka Connect for Confluent Platform - Confluent's documentation on Kafka Connect, the tool for defining connectors that stream data between Kafka and other data systems
• What Is Change Data Capture (CDC)? - Confluent's overview of CDC, including push vs. pull delivery, polling-based timestamp methods, and log-based capture in real-time
• Debezium Features - How Debezium's Kafka Connect source connectors capture every data change with very low delay, unlike polling