Managed stream and batch data processing. Perfect for building real-time analytics pipelines from event data.
Dataflow solves a specific problem: transforming event streams in real time with complex logic that BigQuery's streaming inserts and scheduled queries can't handle. If you need custom sessionisation with configurable timeouts, real-time event enrichment from multiple data sources, deduplication with exactly-once semantics, or windowed aggregations with watermark handling — Dataflow is the correct tool. If you don't need these capabilities, it's expensive complexity.
The Apache Beam programming model is simultaneously Dataflow's greatest strength and highest barrier to entry. Writing a Beam pipeline requires understanding PCollections, Transforms, windowing strategies, triggers, and watermarks. The mental model is fundamentally different from batch SQL. For teams with stream processing experience, this is natural. For analytics teams whose primary skill is SQL, the learning curve is 2-4 weeks of productive time.
Real-time sessionisation is the canonical analytics use case. Define a session window with a 30-minute gap (or whatever your business requires). Events are grouped into sessions as they arrive. Session-level metrics — duration, page count, entry page, exit page, conversion flag — are computed in real time and written to BigQuery. This gives you live session-level analytics without the next-day processing delay of BigQuery's batch sessionisation.
Event enrichment at the stream level is operationally powerful. As events arrive, you can join them with dimension tables (product catalogues, user profiles, geo databases), validate schema compliance, filter bot traffic using IAB lists, and deduplicate using event fingerprints — all before data reaches your warehouse. The enriched events that land in BigQuery are cleaner and more analytically useful than raw client-side data.
Cost modelling for Dataflow requires careful analysis. A streaming pipeline runs 24/7. On Cloud Run or Dataflow Flex Templates, minimum cost is approximately €100-300/month for a simple pipeline. Complex pipelines with multiple transforms and high throughput can reach €500-2000/month. Compare this against BigQuery scheduled queries (which cost only the bytes scanned) plus streaming inserts (which cost ~$0.01/200MB). For many workloads, the batch approach is 5-10x cheaper.
Decision framework: use Dataflow when your latency requirement is under 5 minutes AND your transformation logic requires windowing, enrichment, or exactly-once deduplication that can't be expressed in BigQuery SQL. For everything else, BigQuery scheduled queries run every 15 minutes and cover 90% of 'real-time' analytics needs at a fraction of the cost and complexity.
When it's the right tool
True real-time stream processing with exactly-once semantics
Apache Beam portability across multiple execution engines
Complex windowing and sessionisation logic in streaming context
Automatic autoscaling — handles traffic spikes without configuration
Native integration with Pub/Sub, BigQuery, and Cloud Storage
Cost-benefit reality
Continuous streaming cost of €100-2000/month depending on complexity
Apache Beam learning curve is significant for SQL-native teams
Debugging streaming pipelines requires specialised tooling and skills
BigQuery scheduled queries handle 90% of 'real-time' needs more cheaply
Over-engineering risk is high — most analytics teams don't need true streaming
The verdict
Reserve for genuine sub-5-minute latency requirements with complex transformation logic. For everything else, BigQuery scheduled queries plus streaming inserts deliver adequate real-time capability at 5-10x lower cost and complexity.
Author
Viktor Kvant
Senior Web Analyst specialising in data ownership, advanced tracking implementations, and translating data into strategic business value.