“Should we use streaming or batch?” is a question that gets asked with less precision than it deserves. Both terms describe how data moves through a pipeline, but the right answer depends on requirements that teams often haven’t explicitly defined: how fresh does the data need to be, how much complexity can they operate, and what happens when things fail?

The answer is rarely “streaming is better” or “batch is better.” It’s “what does this use case actually require?” - and that question has a business answer before it has a technical one.

The first question is not “which tool?”. It’s “what does stale data actually cost?” If the fraud detection system runs an hour after a transaction instead of in real time, what is the expected loss from fraud that would have been caught? If the analytics dashboard updates daily instead of hourly, does a business decision get made on wrong information, or does anyone even look between updates? If the answer to these questions is “it doesn’t actually matter much,” then the simpler system is the right one.

What Batch Processing Is

Batch processing runs a computation over a bounded, finite dataset on a schedule. You collect data, process it all at once, write the results.

A nightly ETL job that reads all orders created in the last 24 hours, computes sales metrics, and writes them to a data warehouse is batch processing. So is a weekly report generation job. So is a daily ML model training run.

Tools: Apache Spark, dbt (for SQL transformations), cron jobs with SQL queries, Apache Airflow orchestrating Python scripts.

Properties of batch:

• Processes data at rest (already stored)
• Runs periodically (hourly, daily, weekly)
• Typically handles large data volumes efficiently (optimized for throughput over latency)
• Simple to reason about: you know exactly what data was processed
• Easy to rerun: if it fails or produces wrong results, run it again from the same input

The latency is the batch interval. A daily batch job produces data that’s up to 24 hours stale. For many use cases, this is fine.

What Stream Processing Is

Stream processing operates on data continuously as it arrives. Events are processed as they’re produced, typically with latency measured in milliseconds to seconds.

A fraud detection system that evaluates every transaction as it happens is stream processing. So is a dashboard that updates in real time, or an event-driven pipeline that triggers notifications immediately when a user completes a purchase.

Tools: Apache Kafka (message streaming), Apache Flink (stateful stream processing), Kafka Streams, Spark Structured Streaming, AWS Kinesis.

Properties of streaming:

• Processes data in motion (as it arrives)
• Continuously running
• Low latency (results available seconds after events occur)
• More complex to operate (windowing, watermarks, late data, stateful operations)
• Harder to reprocess historical data (requires replaying from the source log)

The Real Tradeoffs

The streaming vs batch choice is fundamentally about latency requirements vs operational complexity.

Latency: batch processing is limited by its schedule. The fastest batch processing (minute-level or sub-minute jobs) approaches streaming in latency but with more overhead. If you need data to be available within seconds of an event, you need streaming. If hourly or daily freshness is acceptable, batch is simpler.

Complexity: streaming systems are harder to build and operate. State management, event ordering, late-arriving data, exactly-once semantics - these problems don’t exist in batch processing because you process a complete dataset after all the data has arrived.

Fault tolerance: a failed batch job can be rerun from the beginning. A failed streaming job needs to resume from a checkpoint without losing or duplicating events. Getting this right requires careful design.

Testing: batch jobs are deterministic given the same input. Streaming pipelines are harder to test because the inputs are continuous and unbounded.

Cost: streaming requires continuously running infrastructure. Batch jobs run for a period and stop. For infrequent processing, batch is cheaper.

Windowing: Where Streaming Gets Complicated

Streaming becomes complex when you need aggregations over time - “how many users signed up in the last hour?” This requires windowing: grouping events into time-based buckets.

Tumbling windows: fixed, non-overlapping intervals. Events in the 12:00-13:00 window, events in the 13:00-14:00 window. Good for periodic aggregates.

Sliding windows: a window that advances continuously. The last 60 minutes of events, recalculated every minute. More useful for moving averages but more expensive.

Session windows: windows that close when there’s a gap in activity. User sessions are the canonical example - a session ends after N minutes of inactivity.

The tricky part: late-arriving events. A user’s event might be generated at 12:58 but arrive at the processing system at 13:02, after the 12:00-13:00 window has closed. Stream processors use “watermarks” to define how late events can arrive before a window is finalized. Setting the watermark too tight means you miss late events; too loose means higher latency.

This problem doesn’t exist in batch processing. When you run a daily job, all events for that day have already arrived by the time you process them.

The Lambda and Kappa Architectures

Two architectural patterns address the streaming/batch tradeoff:

Lambda architecture: run both a batch layer (accurate, delayed) and a streaming layer (fast, approximate). Merge results to serve queries. Solves the problem of having both freshness and accuracy, at the cost of maintaining two separate pipelines.

Kappa architecture: use streaming as the only layer, but with a reprocessing capability - replay the event log through the streaming system to recompute historical data when needed. Simpler than Lambda (one system), but requires a replayable event log (Kafka with sufficient retention).

In practice, many teams discover they built a Lambda architecture accidentally (a real-time feed and a nightly job that corrects it) and migrate toward Kappa as their streaming system matures.

Micro-Batching

Some systems (Spark Structured Streaming, early versions) achieve near-real-time processing by running tiny batch jobs every few seconds. This is “micro-batching” - a middle ground that provides batch processing’s simplicity with latency in the seconds range.

It’s a pragmatic choice for teams that know batch processing well and need lower latency than a daily job. The Flink and Kafka Streams model (true streaming) provides lower latency but with more operational complexity.

A Decision Framework

Start with latency requirements:

• Need results within seconds of an event: streaming
• Need results within minutes: micro-batching or small-interval batch
• Need results within hours or days: batch

Then assess operational capacity:

• Small team, limited data engineering expertise: batch (significantly simpler to operate)
• Team with streaming experience, mature observability: streaming is viable

Then consider data volume and access patterns:

• Very large datasets processed infrequently: batch optimizes for throughput
• High-event-rate continuous data: streaming is the natural fit

Operational Cost

The operational cost difference between streaming and batch is substantial and often underestimated before teams have run both.

A batch job has a lifecycle: it starts, runs, finishes, and stops. When it fails, you look at the logs, fix the issue, and rerun it. The state is bounded: the input data, the output data, and the job code.

A streaming system is always running. When a streaming pipeline fails, it might fail silently - processing events incorrectly rather than stopping. Or it might fall behind, accumulating a lag that grows until the system can’t catch up. Or it might restart and reprocess events, producing duplicates if the pipeline isn’t idempotent. The operational surface is much larger: consumer group lag monitoring, exactly-once semantics, checkpointing and replay, watermark tuning for late data.

In practice: a streaming system requires more engineering time to operate, more observability investment to monitor correctly, and deeper expertise to debug when something goes wrong. For a two-person data team, this can consume most of the available capacity. For a team with dedicated data engineers and existing streaming infrastructure, it’s manageable.

The honest cost breakdown: streaming systems cost more to build, more to operate, and more to debug. They earn that cost back in lower data latency. If your latency requirement is “hours or less,” the question is whether the operational investment is worth the latency improvement you actually need.

The mistake most teams make is defaulting to streaming because it sounds modern, then spending months dealing with complexity that wasn’t necessary for their actual latency requirements. Batch processing is underrated. It’s simple, reliable, and correct for more use cases than people admit.