An end-to-end stream and batch data processing pipeline built for a leading commercial vehicle manufacturer in India. The system ingested, processed, and analyzed real-time sensor data from trucks and farm vehicles — capturing everything from engine diagnostics and fuel consumption to GPS coordinates and driver behavior.
Originally designed for a fleet of 10,000 vehicles, the architecture scaled horizontally without any structural changes to support hundreds of thousands of vehicles and tens of millions of sensor data points.
The streaming layer was powered by Google Pub/Sub and Cloud Dataflow, enabling real-time ingestion and processing of high-throughput sensor telemetry. A dynamic, config-based streaming alerts module handled aggregation logic, making it straightforward to define and modify alert rules without code changes.
On top of this, an alerting system was built using windowing functions for time-based aggregations, geofencing for location-aware triggers, and trip and driver shift detection for operational insights — all evaluated on live streaming data.
For the batch layer, processed data landed in BigQuery for historical analytics and reporting, while BigTable served as the low-latency store for time-series lookups. Redis handled caching and fast state access for the streaming pipeline.
The clean separation between stream and batch paths meant each could be tuned independently. The config-driven alert design allowed operations teams to roll out new monitoring rules across the entire fleet without engineering intervention.