Accidentally deleting data in production is often seen as a rite of passage in the development community. Downtime is the war story of our industry. I'm collecting those stories in a series called "This Is Fine".
You can listen to my story here:
Here's the written version:
I was a young PHP developer at an IT computer college in South Africa. We had a bug in production for a specific user that I was trying to reproduce. To do this, I connected to the production database, located the row, and duplicated it in my development database. Doing this allowed me to execute the code on my machine and create a reproduction.
After adding a few print statements, I was able to identify the bug. Fixing it required making a few changes, running through a workflow and observing the end result. Each time I made changes. I had to reset the state of my database. I was lazy, so I simply deleted all the users and reinserted the problematic row. And then it was lunchtime. After a heavy meal, I returned to my desk and reconstructed my mental model of the bug I was fixing. I went to the terminal and reset my database state. Instead of the prompt returning in a few milliseconds. It took much longer, far longer than it should have. Realization struck and a wave of cold washed over me. I had accidentally executed my statement on the production database.
I tried to interrupt the statement by smashing ctrl+c, but it was too late. The data was gone. And most likely my job too. I knew I couldn't hide this. So I turned my chair around and looked my boss straight in the eyes–like a deer in the headlights."What's going on?" he asked. "I accidentally deleted the user's table in production!"
We put up a "things are on fire page" to inform our customers that I had caused a major outage, and then restored the backup from earlier that morning. All in all, it took about 30 minutes to bring things back online and our students lost a few hours of progress. My nerves were shot and I felt ashamed. But my boss calmed me down: "It happens to everyone, and hey, we had backups. So it wasn't as bad as the time I deleted production."
We chatted a bit about his war story and then created a read-only user.
If you've deleted data in production, I want to hear from you! Your story will earn the coveted 'This is Fine' badge, and your failure will be memorialized forever on the memory of the Internet.
