This blog has had a pretty good run as far as data retention goes. I’ve been through probably a dozen different servers over its life and every time I’ve managed to maintain continuity of pretty much everything. It’s not because I kept rigorous backups or anything like that, no I was just good at making sure I had all my data moved over and working before I deleted the old one. Sure there’s various bits of data scattered among my hard drives but none of it is readily usable so should the unthinkable happen I was up the proverbial creek without a paddle.
And, of course, late on Saturday night, the unthinkable happened.
So I logged into my blog to check out how everything was going (as I usually do) and noticed that something strange was appearing in my header. It appeared to be some kind of mass mailer although it wasn’t being pulled in from a JavaScript file or anything and, to my surprise, it was embedding itself everywhere, even on the admin panel. Now I’ve never been compromised before, although people have tried, so this sent me into something of a panic and I started Googling my heart out to find out where this damn code was coming from. Try as I might however I couldn’t find the source of it (nothing in the Apache configuration, all WordPress files were uncompromised, other sites I’m hosting weren’t affected) and I resigned myself to rebuild the server and to start anew. Annoying, but nothing I haven’t done before.
Like a good little admin I thought it would be good to do a cleanup of the directory before I embarked on this as I was going to have to move the backup file to my desktop, no small feat considering it was some 1.9GB big and I’m on Australian Internet (thanks Abbott!). I had a previous backup file there which I moved to my /var/www directory to make sure I could download it (I could) and so I looked to cleaning everything else up. I’ve had a couple legacy directories in there for a while and so I decided to remove them. This would have been fine except I fat fingered the command and typed rm -r which happily went about its business deleting the entire folder contents. The next ls I ran sent me into a fit of rage as I struggled to figure out what to do next.
If this was a Windows box it would’ve been a minor inconvenience as I’d just fire up Recuva (if CTRL + Z didn’t work) and get all the files restore however in Linux restoring deleted files seems to be a right pain in the ass. Try as I might extundelete couldn’t restore squat and every other application looked like it required a PhD to operate. The other option was to contact my VPS provider’s support to see if they could help out however since I’m not paying a terrible amount for the service I doubt it would been very expedient, nor would I have expected them to be able to recover anything.
In desperation I reached out to my old VPS provider to see if they still had a copy of my virtual machine. The service had only been cancelled a week ago and I know a lot of them keep copies for a little while just in case something like this happens, mostly because it’s a good source of revenue (I would’ve gladly paid $200 for it). However this morning the email came from them stating unequivocally that the files are gone and there’s no way to get them back, so I was left with very few options to get everything working again.
Thankfully I still had the database which contains much of the configuration information required to get this site back up and running so all that was required was to get the base WordPress install working and then reinstall all the necessary plugins. It was during this exercise that I stumbled across the potential attack vector that let whoever it was ruin my site in the first place: my permissions were all kinds of fucked, essentially allowing open slather to anyone who wanted it. Whilst I’ve since struggled to get everything working like it was before I now know that my permissions are far better than they were and hopefully should keep it from happening again.
As for the rest of the content I have about half of the images I’ve uploaded over the past 5 years in a source folder and, if I was so inclined, could reupload them. However I’ve decided to leave that for the moment as the free CDN that WordPress gives you as part of Jetpack has most of those images in it anyway which is why everything on the front page is working as it should. I may end up doing it anyway just as an exercise to flex my PowerShell skills but it’s no longer a critical issue.
So what has this whole experience taught me? Well mostly that I should practice what I preach as if a customer came running to me in this situation I’d have little sympathy for them and would likely spend maybe 20% of the total effort I’ve spent on this site to try and restore theirs. The unintentional purge has been somewhat good as I’ve dropped many of the plugins I no longer used which has made the site substantially leaner and I’ve moved from having my pants around my ankles, begging for attackers to take advantage of me, to at least holding them around my waist. I’ll also be implementing some kind of rudimentary backup solution so that if this happens again I at least have a point in time to restore to as this whole experience has been far too stressful for my liking and I’d rather not repeat it again.
This one time I shift-deleted my music collection and did not have a recent backup. I caught the process pretty quickly though! I have no idea what is missing.
Good to see you are back online.
Heh my music collection met death by the hand of my cat, thinking the external hard drive cord was something fun to play with.
And you’d think after that data disaster I would’ve learnt my lesson. It seems I’m something of a slow learner 😉
Glad to see you’re still up here and operational. =)
Cheers 🙂 I considered just not doing it at one point but I just couldn’t bring myself to shut it down after all these years.
[…] Last year I fucked up. […]
I am laughing at you a bit right now. I can see how the person who came up with the default help entry must have had a serious sense of humour; I know -r to mean recursively, as in this directory and all directories in it; but the help entry certainly makes it sound like it only removes subdirectories XD.
Yeah looking back on it now it was pretty comical as I honestly should have made the full backup before I attempted any kind of cleanup or maintenance. I’m now on a much better provider who handles all that kind of stuff for me (including security!) so now I’m at least a couple steps removed from being able to conduct a repeat performance.
I did actually end up running a script to restore most of the images and I think I’ve gotten it down to a handful I can’t find which I think is pretty good considering I didn’t do any formal backup at all. Not that I’d rely on the methods I used to get stuff back though (it’s a combination of local files with a lookup to the Internet Archive) but I was still impressed at the recovery rate I was able to get.