One Line of Code Accidentally Deletes Entire Company

As far as code mistakes go, few can claim that their careless coding practices caused the deletion of their entire company. Marco Marsala ran a small web hosting company that carried the websites of a number of clients until he unwittingly instructed the servers to delete their entire contents, effectively wiping out his business and the websites of his clients.

In response to the tragedy that befell his servers, Marsala took to the Server Fault forum to explain his plight and perhaps hope that some of the forum’s denizens would be able to help him with his predicament. Instead of help, most of the advice he received simply informed him that the chance he had forever deleted his company was high and his code had completely destroyed both his own data and that of his clients.

I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.

The reason why Marsala lost all of his data stems from his use of the “rm -rf” command, which can be broken down to “rm”, removing files, “-r” meaning it will delete recursively into every subfolder and “-f” for force, meaning no warning will be given. Due to the two variables surrounding the / being empty, this caused the system to delete from the root directory, essentially wiping out everything on the machine. To make matters worse, while he had taken backups, the backup devices had been mounted just before the erroneous script ran, causing them to also be wiped.

Responses to Marsala’s post ranged from pity to insulting, however, all agreed that the data on the servers was almost certainly gone for good with no recovery. Most focused on pointing out the mistakes he had made, instead of being able to offer him any help, “This is not bad luck: it’s astonishingly bad design reinforced by complete carelessness” wrote user Massimo.

For Marsala, there doesn’t look to be a good end to this story. There are very few options open to him that would allow the data to be recovered and even those, such as contacting professional data recovery experts, are expensive, time-consuming and have no guarantee of success. This should serve as a cautionary tale for those wishing to start their own online businesses to be very careful over what you run on your servers and the care you take of your backups.