Snappy, the Snaplet cat mascot trembles at the thought of acidentaly deleting a production database

“This is fine”–That time I deleted our production database by mistake

Peter Pistorius, Snaplet CEO, recalls his first nerve-wracking experience as a young developer.

Accidentally deleting data in production is often seen as a rite of passage in the development community. Downtime is the war story of our industry.  I'm collecting those stories in a series called "This Is Fine".

You can listen to my story here:

Here's the written version:

I was a young PHP developer at an IT computer college in South Africa. We had a bug in production for a specific user that I was trying to reproduce. To do this, I connected to the production database, located the row, and duplicated it in my development database. Doing this allowed me to execute the code on my machine and create a reproduction.

Accidentally deleting your production database...

After adding a few print statements, I was able to identify the bug. Fixing it required making a few changes, running through a workflow and observing the end result. Each time I made changes. I had to reset the state of my database. I was lazy, so I simply deleted all the users and reinserted the problematic row. And then it was lunchtime. After a heavy meal, I returned to my desk and reconstructed my mental model of the bug I was fixing. I went to the terminal and reset my database state. Instead of the prompt returning in a few milliseconds. It took much longer, far longer than it should have. Realization struck and a wave of cold washed over me. I had accidentally executed my statement on the production database.

I tried to interrupt the statement by smashing ctrl+c, but it was too late. The data was gone. And most likely my job too. I knew I couldn't hide this. So I turned my chair around and looked my boss straight in the eyes–like a deer in the headlights."What's going on?" he asked.  "I accidentally deleted the user's table in production!" 

We put up a "things are on fire page" to inform our customers that I had caused a major outage, and then restored the backup from earlier that morning. All in all, it took about 30 minutes to bring things back online and our students lost a few hours of progress. My nerves were shot and I felt ashamed. But my boss calmed me down: "It happens to everyone, and hey, we had backups. So it wasn't as bad as the time I deleted production."

We chatted a bit about his war story and then created a read-only user.

If you've deleted data in production, I want to hear from you! Your story will earn the coveted 'This is Fine' badge, and your failure will be memorialized forever on the memory of the Internet.

Peter Pistorius
June 22, 2021