The 50x Multiplier: How My Fan-Out Architecture Almost Blew the Budget

I’ve always thought the Pub/Sub Fan-Out pattern was a cool concept. It’s a classic way to decouple services: you publish one message to a topic and it triggers multiple subscribers at once. It’s elegant and reactive, so I figured I’d give it a shot for my background jobs.

It works great when everything is smooth. But I recently learned that if you don't watch your failure states, a Push-based fan-out scales your bill faster than your app.

The Math of the Mess: 50+ Concurrent Calls

The setup was straightforward. One message entered a Pub/Sub topic and fanned out to my four primary background jobs. Each subscription triggered its own Cloud Function.

On a good day, it’s a clean flow. But when a downstream dependency failed, I ran into a frustrating GCP limitation. Google requires a minimum of 5 delivery attempts for Push messages before they can be sent to a Dead Letter Queue. This means a single failure triggers a mandatory, aggressive retry loop.

The distribution wasn't even, either. Because of the way the data was skewed and how the retries hit the system, the concurrency was all over the place. Some jobs stayed stable while others spiraled. I ended up with over 50 concurrent Cloud Function calls spiking at once. I wasn't just paying for a bug; I was paying for that bug to be repeated dozens of times in a frantic, lopsided burst of compute.

Returning to the Basics (and UNIX)

I realized pretty quickly that for what I’m doing right now, "Real-Time" was a luxury I didn't need and a risk I couldn't afford. I needed a way to trigger these processes without the danger of an expensive billing loop.

I decided to rip out the Pub/Sub topic and replace it with something much more predictable: Cloud Scheduler.

There is a bit of irony in making this move. My first experience with job scheduling was decades ago, writing SAS code in a UNIX environment. Back then, we relied on Cron because it was robust and did exactly what it was told. Returning to a Cron-based model in GCP felt like returning to my roots.

By moving to this batch model, I solved the problem instantly:

• Predictable Execution: Instead of pushing data the second it arrives, Cloud Scheduler wakes up my functions on a fixed schedule. I went from erratic spikes to a predictable window. I know exactly how many times my functions will run each month.
• Absolute Control: Unlike Pub/Sub, Cloud Scheduler lets me set retries to zero. If a function fails because a 3rd-party API is down, it just logs the error and stops. It doesn't panic or retry five times. It just waits for the next scheduled run.
• Cost Stability: My cloud bill is back under control. Cloud Scheduler is cheap, and my function invocations are now a flat line on a graph instead of a series of expensive spikes.

The Takeaway: Real-Time isn't Always Right

As engineers, we often chase the most "reactive" solution because it’s a fun challenge. But elegant architecture can be a nightmare to manage during a failure.

For this use case, a 15-minute delay was a small price to pay for 100% predictability. Sometimes the "old school" methods from the UNIX days are still the most reliable tools in the shed.