Why pipelines break at 3am
It is almost never the code. Here is what actually goes wrong, and the three habits that stopped most of my night pages.
I carried on-call for an ingestion platform for about three years. In that time I got paged at every unpleasant hour you can name. Here is the thing nobody tells you when you start: the code you wrote is almost never the problem.
The code ran fine yesterday. It ran fine for the last two hundred days. What changed was everything around it.
The four things that actually break
Someone upstream changed a payload. A team you have never met added a field, renamed a column, or switched a date format from ISO to something regional. They did not tell you because they did not know you existed. Your job failed at 3:04am because a parser hit a string where it expected a number.
A file did not show up. The vendor’s job failed, or their SFTP box filled up, or a holiday you did not know about shifted their schedule. Your pipeline is sitting there waiting for something that is not coming.
The data got bigger. Slowly, then suddenly. A partition that used to hold two million rows now holds forty million because marketing launched something. Your Spark job that finished in twenty minutes now runs for three hours and hits the window where the next run starts.
Something retried into a mess. The first attempt half wrote. The retry ran on top of it. Now you have duplicate rows and a downstream report that says revenue doubled.
Notice that none of these are bugs. They are all the world changing underneath a thing that assumed it would not.
What actually helped
Data contracts, even informal ones
At T-Mobile we kept getting broken by unannounced payload changes. The fix was not clever code. It was a schema registry and an agreement with the producing teams: if you change the shape, you version it, and the old version keeps working for a deprecation window.
That one change ended an entire category of pages. Not reduced it. Ended it.
The hard part was not technical. It was walking over to four other teams and having the conversation. Worth it.
Alert on symptoms, not on noise
Early on I alerted on everything. Job failed, page me. Job ran long, page me. Row count moved, page me.
What happens is you stop reading them. When everything is urgent nothing is, and you start swiping the page away at 3am while half asleep, which is how a real incident gets missed.
Now I only page on things that mean a person downstream is about to be wrong:
- The table a report depends on is stale past its promise
- Row counts moved outside a band that has held for months
- A job failed its last retry, not its first
Everything else goes to a channel I read in the morning.
Make the retry safe
If your job cannot be run twice safely, you do not have a pipeline, you have a ritual. Idempotent writes, partition overwrites instead of appends, exactly once where the platform supports it. Then a retry is just a retry, and you can let the system fix itself before it ever reaches your phone.
The honest summary
Uptime went from 97% to 99.5% on that platform. Almost none of that came from writing better transformation logic. It came from assuming things would change without warning and building for it.
The pipeline is the easy part. The world around the pipeline is the job.