Portfolio / Flash Crash Detector

Real-Time 'Flash Crash' Detector

Project Summary

Built an end-to-end streaming pipeline that ingests live Bitcoin price data polled from the Binance REST API, publishes those events to Google Pub/Sub, and processes them through an Apache Beam pipeline running on Dataflow. Every 60 seconds, the pipeline emits a windowed summary record to BigQuery containing the min, max, average, and drop percentage, the total trade count, and a boolean flash_crash_flag that is set to true when the within-window price drop percentage meets or exceeds 0.25%.

A key design constraint was that the fully cloud-hosted ingestion path wasn't viable due to cost and IP address restrictions on the Dataflow side. The solution used a local Python publisher as the Pub/Sub producer, with all processing and storage remaining on Google Cloud — a hybrid architecture that kept the cloud footprint lean while preserving the streaming semantics.

Key Requirements

• Continuously poll live BTC prices from the Binance REST API.
• Publish events to Pub/Sub and process with Apache Beam on Dataflow.
• Aggregate per 60-second tumbling window: min, max, avg, drop price, trade count.
• Set a flash_crash_flag for any window where the price drop exceeds 0.25%.
• Write results to BigQuery for querying and historical analysis.

Architecture

A hybrid pipeline — local producer feeding a fully managed Google Cloud processing stack.

Step 1

Price Producer

A local Python script polls the Binance REST API for live Bitcoin prices and publishes each event to a Google Pub/Sub topic.

Step 2

Pub/Sub

Google Pub/Sub decouples the local producer from the cloud layer, acting as a durable, ordered message buffer for the incoming price stream.

Step 3

Dataflow

Apache Beam on Dataflow reads from Pub/Sub, applies 60-second fixed windows, computes price metrics, and evaluates the flash crash threshold.

Step 4

BigQuery

Each windowed record — including the flash_crash_flag — is written to BigQuery for querying and historical analysis.

Pipeline Output Schema

Every 60-second window produces one row in BigQuery with the following fields.

BigQuery — flash_crash_events

Field	Type	Description
window_start	TIMESTAMP	Start of the 60-second tumbling window
min_price	FLOAT	Minimum BTC price observed in window
max_price	FLOAT	Maximum BTC price observed in window
avg_price	FLOAT	Average BTC price across all events in window
drop_pct	FLOAT	((max_price − min_price) / max_price) × 100
trade_count	INTEGER	Number of price events received in window
flash_crash_flag	BOOLEAN	true when drop_pct > 0.25

Engineering Decisions

Key trade-offs made during the build.

Hybrid Architecture

Running the Binance API producer locally and bridging to Pub/Sub was the pragmatic call when cloud-hosted ingestion wasn't viable due to IP restrictions and cost. Pub/Sub absorbed the boundary cleanly without changing the downstream processing at all.

60-Second Tumbling Window

Fixed 60-second windows provide a clean, non-overlapping unit of analysis. Each window produces exactly one BigQuery row, making downstream SQL queries deterministic and easy to reason about without session or sliding window complexity.

Explicit 0.25% Threshold

The drop percentage is computed as (max − min) / max × 100 per window. The flash_crash_flag is an explicit boolean rather than embedded logic, so the sensitivity can be tuned in SQL without redeploying the Dataflow job.