Streaming Data Pipelines:
When You Actually Need Them

The Streaming Trap

Real-time data streaming is genuinely transformative for the problems it’s designed to solve. But a large share of the data pipelines we encounter in the wild didn’t actually need any of that. They needed a reliable, well-monitored batch process that ran every fifteen minutes. The engineering team chose Kafka because it felt like the right call for a “data-driven company,” and they’re now operating a distributed streaming infrastructure that requires specialist knowledge to maintain.

The right streaming architecture question is not “should we use Kafka?” it’s “what’s the actual latency requirement for this use case, and what’s the cost of getting it wrong?”

When Streaming Is the Right Answer

Streaming makes sense when your business logic genuinely requires near-real-time processing typically sub-minute latency and when the volume or velocity of data makes polling or micro-batch approaches impractical. Fraud detection that needs to act on a transaction within 200 milliseconds is a legitimate streaming problem. IoT sensor data from ten thousand devices that needs to be processed continuously is a streaming problem. Live inventory systems that need to reflect warehouse reality without a lag that causes overselling are streaming problems.

These use cases share a common characteristic: the cost of latency is directly measurable in business terms. Fraud detected after 30 seconds is fraud undetected. Inventory state that’s 15 minutes stale causes real overselling incidents. That business cost is what justifies the engineering complexity of a streaming architecture.

When Batch Is the Right Answer

Batch processing is underrated, largely because “real-time” has become a status signal in engineering rather than a technical requirement. Most reporting pipelines don’t need to be real-time. Most recommendation systems can tolerate hourly updates. Most analytics dashboards are fine updating every 15 minutes. If the business impact of a 15-minute data delay is negligible and it usually is for analytics use cases batch is the right answer.

The advantages of batch over streaming are significant: easier to reason about, easier to test, easier to debug when something goes wrong, dramatically cheaper to operate, and much simpler to backfill when you discover a bug in your processing logic. A dbt pipeline running on Airflow handles the vast majority of analytics data needs at a fraction of the operational complexity of Kafka.

The Latency Requirement Framework

Before choosing an architecture, answer these questions precisely:

• What is the maximum tolerable latency for this use case? Not “as fast as possible” an actual number in seconds or minutes.
• What happens when that latency is exceeded? Lost revenue, operational incident, customer complaint, or nothing significant?
• What is the data volume and velocity? Events per second, GB per hour, peak vs average throughput.
• Does the processing logic require state across events? Windowed aggregations, sessionisation, and joins across streams require stateful processing engines.

If your maximum tolerable latency is above 5 minutes and your data volume is below 1GB/hour, you almost certainly don’t need Kafka. A micro-batch approach Spark Structured Streaming, Flink in micro-batch mode, or even a well-tuned Airflow DAG will serve you far better.

Choosing the Right Stack

For genuine streaming requirements, the choice between Kafka, Kinesis, and Pub/Sub is largely determined by your cloud provider and operational preferences. Kafka is the most flexible and portable but requires the most operational investment. Kinesis is the lowest-friction choice for AWS shops. Pub/Sub is the natural choice on GCP.

For stream processing logic, Flink is the most capable engine for complex stateful processing and is increasingly the industry standard for serious streaming workloads. Spark Structured Streaming is a reasonable choice if your team already knows Spark. ksqlDB is useful for simpler SQL-based stream transformations on Kafka. For micro-batch and scheduled pipelines, dbt on Airflow or Prefect handles the majority of analytics use cases with minimal operational overhead.

The Migration Path

The teams that handle this best start with the simplest architecture that meets their current latency requirements, instrument it carefully, and scale up to streaming when the data shows they need it not before. Build your batch pipeline first. Measure actual latency against business requirements. Only when you have clear evidence that batch latency is causing real business impact should you invest in a streaming architecture.

This approach also gives you a fallback: a well-tested batch pipeline can serve as a recovery mechanism when your streaming pipeline has an incident, which is a question worth considering before you remove your batch processing entirely.