On Friday at 3:54pm Eastern, the FAA stopped every departing flight in the US. It lifted the order at 8:15pm. More than 7,000 flights were cancelled, NBC New York reported.
Most of the coverage asks why the system is so old. Fair question. It is not the one I would ask.
Here is what officials said. Transportation Secretary Sean Duffy blamed an outdated system where overloaded routers caused instability in radar and communications. He said the FAA did not lose all radar or telecom, and that the stop was a precaution. The Homeland Security Secretary said there was no sign of a cyberattack, though it could not be fully ruled out. NBC News describes a router problem at one FAA site. Headlines in the Wall Street Journal and the Washington Post say Duffy linked it to work replacing old wiring.
I can't tell you which explanation is complete. The reports are still coming in. What I can describe is the shape of the failure: trouble first spotted at 3:36pm, and a whole country stopped within the hour.
That shape is my question. How far does one failure spread? Engineers call it the blast radius. A small one means a problem in one place hurts a few people. A large one means it hurts everyone, even when the broken part is small.
If the wiring reports are right, there is a second lesson. Upgrades are where outages hide. The work that makes a system safer is also the moment it is most likely to break.
The FAA chose to stop everything as a precaution. That is a defensible call when you cannot see the edges of a problem. It is also what happens when the walls were never built.
For your own product, here is the one number I would work out this week:
customers hurt by one failed part / all customers
Do it for your database, your login service, your payment provider and your hosting region. If any answer is close to 1, that part is your router. Whichever number is closest to 1 gets your attention first, and any upgrade work on it gets a rollback plan.
Written by
Lars ErikssonFounder, Flowmetric20 pieces · 13 followersFollow Lars to see new posts in your feed, on your phone or by email.
More from
Lars Eriksson →If CrowdStrike buys Zscaler, what happens to your contract?
Bloomberg reported on Friday that CrowdStrike and Zscaler are in talks about a deal, citing people familiar with the matter. No agreement has been announced. CrowdStrike declined to comment, and CNBC…
How Microsoft bills its new Copilot agent: a seat plus a meter
More than 80% of corporate AI spending now sits outside the IT budget, and total spending doubled in a year. That is Microsoft's claim, citing BCG, in a blog post on 17 November. I have not seen the B…
Who can edit the scripts on your checkout page?
Zoom, the BBC, CNN, Salesforce, GitLab and HSBC all served the same malicious script on Monday. None of them wrote it. Report URI, a security firm, checked 1,611 Optimizely snippets and found 188 inf…