Accidentally deleting the users table — and you can too!

Pictured: disappointment. Photo by Sarah Kilian on Unsplash

Let me tell you about the time I accidentally deleted the users table. Why? Well people make mistakes all the time andI think it’s important to be open about them so they get fixed quickly. I think it’s important to normalize talking about mistakes because they are a part of learning, we all make them, and they are hilarious. So gather round….

We, like most businesses, have a users table. It has, as you may have guessed, data for all of the users of Nested. At the time of deletion the table was moderately large - not huge but several thousand rows, each with varying amounts of information depending on how far through the funnel a user got with Nested.

Now when a new internal user joins Nested, they sign up for an account and we give them some extra permissions. At the time of deletion one of the ways to do this was to shell into a production server and run a rake command. What’s a rake command? Well the server we would shell into was a small ruby app. Rake is a ruby library that lets you run tasks (that you define) on the app.

As you may have figured out when you heard “shell into a production server” this isn’t the most bombproof way to set things up, but being a startup sometimes you go with what works until there is a dire need to fix it.

Defining a rake task to take care of promoting users to internal (and therefore privileged users) is also slightly better than, for example, shelling into a production database and running some arbitrary SQL. Imagine, for example, if the sql you needed to run was this:

UPDATE users SET admin=true WHERE user_id='ABCD123'

now imagine you had connected to the production database and forgot to add the where clause. Or worse typo’d this:

UPDATE users SET admin=true; WHERE user_id='ABCD123'

Anyway, back to the story. So I got a shell on a production server, which had rake tasks on it. Now to be especially sure I didn’t make a mistake and run the wrong rake task I did the following:

bundle exec rake -t

That lists all of the available rake tasks in the app.

Except it doesn’t.

The actual command is:

bundle exec rake -T

rake -t is rake —trace which runs the default rake task, logging output to the console. Now, in an unrelated foo bar we also had this in our docker file:

RUN bundle install --retry 10 --jobs 4

That installs the project dependencies. However what we should have had is:

RUN bundle install --retry 10 --jobs 4 --without development test

That installs the project dependencies, leaving out the ones that are only needed for test and development. Dependencies like rspec (the ruby testing library) and the database_cleaner, which deletes all the data in the database between tests.

Have you guessed it yet? The default rake task was to run the tests. We didn’t exclude the development and test gems, meaning 1. I could run the tests and 2. when I ran them, rake dutifully wiped the database afterwards. The production database. With all the users in.

Pros and Cons of the situation that led to deleting everything

Pro — There were database backups.

Pro — Very GDPR friendly.

Pro — The app was connected to its own database with not much in it past the users table — meaning limited damage.

Con — There was a bit of a panic when no one could figure out what was wrong.

Pro — It did motivate some changes and improvements to how we do things.

All in all I give it a 2/10, would not delete again.

What did we learn?

Each fuck up is a new chance to learn and understand what needs to be changed to avoid it in the future. Hopefully you can see that it usually takes a litany of errors (well, sometimes just two) for something catastrophic to happen. So next time something goes wrong, try to understand the timeline, what happened and what can be improved to avoid it?

I believe it was John Donne who said:

No fuck up is an island entire of itself; every fuck up is a piece of the continent, a part of the team.