The Failure Ledger Archive
Investigative documentation of corporate blunders, career mistakes, and systemic crashes.
The GitLab Accidental Database Deletion: How Five Silent Backup Failures Magnified a Single Typo
The complete engineering forensic reconstruction of the 2017 GitLab database deletion. How a single engineer executed an erroneous removal command, exposing five distinct, silent failures in the distributed backup and disaster recovery architecture.
The $150 Million Halloween: How a Rushed SAP & Manugistics Rollout Paralyzed Hershey
A forensic analysis of the 1999 Hershey ERP implementation disaster, where an aggressive 30-month timeline, skipped testing phases, and a Big Bang rollout left the company unable to ship candy for Halloween.
Patriot Missile Software Failure: How a 24-Bit Timing Error Caused the Dhahran Disaster
How a 24-bit limited-precision time conversion accumulated over 100+ hours and caused the Patriot system to lose track of an Iraqi Scud at Dhahran in 1991.
The ยฃ330 Million Aftermath: How TSB's Core Banking Migration Failed
When TSB completed a multi-stage migration to Sabadell's Proteo4UK platform in April 2018, the customer data transferred successfully โ to the penny. What followed was a failure of assurance: inconsistent data-centre configuration that extensive testing never exposed, thousands of unresolved defects at go-live, and a platform that collapsed when exposed to the scale and conditions of live customer traffic.
Mars Climate Orbiter: How a MetricโImperial Software Interface Error Caused a $125 Million Mission Loss
The complete engineering forensic reconstruction of the 1999 Mars Climate Orbiter loss. How an unverified software interface contract in the SM_FORCES ground system violated metric specifications, corrupted trajectory modeling, and survived nine months of cruise operations.
Therac-25 Radiation Overdose: How a Race Condition, Counter Overflow, and Missing Hardware Interlocks Caused a Safety-Critical Failure
A forensic reconstruction of the 1985โ1987 Therac-25 radiation overdoses. How undocumented code reuse, a concurrent state-consistency race condition, a 1-byte counter rollover, and the removal of physical hardware interlocks combined in a catastrophic systems engineering failure.
The 16-Bit Integer That Destroyed a $370 Million Rocket
How an unprotected legacy variable conversion caused the Ariane 5 rocket to self-destruct 39 seconds after launch, exposing the lethal danger of unquestioned software reuse.
The $6.2 Billion Risk-Model Failure: Inside JPMorganโs London Whale
In 2012, JPMorgan Chase's Chief Investment Office suffered $6.2 billion in losses on its Synthetic Credit Portfolio amid a breakdown in risk-model governance, manual spreadsheet controls, risk limits, valuation practices, and trading oversight. A flawed new VaR methodology materially understated reported risk.
When the Database Lied: Inside Target Canada's $2.1 Billion Supply Chain Collapse
How corrupted product data, an aggressive launch schedule, and tightly coupled warehouse systems turned a digital data-quality problem into a physical logistics crisis.
The UK Post Office Horizon IT Scandal: When the Computer is Always Right
Fujitsu's buggy Horizon EPOSS software generated massive phantom shortfalls across UK post offices. Armed with the belief that 'the computer is always right,' the Post Office wrongfully prosecuted over 900 subpostmasters.
You Can't Automate Drywall: Inside Zillow's $408 Million Algorithmic House-Flipping Collapse
In late 2021, Zillow shuttered its algorithmic house-flipping division, recording a $304.4 million Q3 inventory write-down ($407.9 million for the year) and reducing its workforce by 25%. Here is the forensic anatomy of a system where prediction throughput outran physical throughput.
The $900 Million Checkbox: How Three People Looked at the Same Screen and Wired an Entire Bank's Money
A confusing enterprise interface, an unselected checkbox triad, and three layers of rubber-stamp approval caused Citibank to accidentally wire $893.9 million to hostile hedge funds. Here is what the evidence actually shows.
He Deleted the Production Database and Let the Intern Take the Fall: Inside a Startup's Midnight Scapegoat Coverup
At 11:42 PM, a senior lead ran DROP DATABASE on production instead of staging, discovering that 6 months of S3 backups were dead. What followed was an anatomy of workplace terror and scapegoating.
The $440 Million Trading Glitch That Killed Knight Capital in 45 Minutes
On August 1, 2012, Knight Capital lost $440 million in 45 minutes when a legacy feature called 'Power Peg' woke from an eight-year sleep and flooded the market with 4 million rogue orders. Here is the forensic post-mortem.
The 1-Yen Stock Order That Broke Tokyo: How Mizuho Securities Sold 610,000 Shares of a Company That Only Had 14,500
On December 8, 2005, a broker typed 'sell 610,000 shares at 1 yen' instead of 'sell 1 share at 610,000 yen'. Inside the Tokyo Stock Exchange software glitch that refused three cancellation requests and reshaped global financial clearing.
The $105 Billion Typo: How Samsung Securities Issued 2.8 Billion Ghost Shares and Triggered a 37-Minute Trading Free-for-All
On April 6, 2018, a clerk entered 1,000 shares instead of 1,000 won. Inside the internal ledger glitch that minted $105 billion in unbacked phantom equity, sparking a 37-minute insider selling frenzy.
The Bedroom Trader Who Helped Trigger the $1 Trillion Flash Crash: Navinder Sarao and the 2010 Market Fracture
On May 6, 2010, the US financial market plunged nearly 1,000 points in thirty-six minutes. Five years later, federal investigators traced a key catalyst to a modest bedroom in suburban London.
The Friday Morning Update That Took Down 8.5 Million Computers: The Anatomy of CrowdStrike's Channel File 291
At 04:09 UTC on July 19, 2024, a 40KB configuration file was pushed to 8.5 million Windows machines. Within minutes, the global economy froze. Here is the forensic systems post-mortem.