Ok, I’m writing this again, hopefully for the last time.
Last Thursday, a RAID controller on the server that hosts this site went bad and took out the entire server’s data. As far as I’ve been able to determine, the site data is completely unrecoverable.
I temporarily put my most recent backup (from March) onto a Dreamhost server and, as expected, I’m consuming 20% of their CPU time. I hear things about paying for overages there, so it seems like a good idea to get out while the gettin’s good.
Unixshell (the original host) provided a new VPS for me to use, and within a day I had a brand new Debian server online. It took the weekend to get things how I wanted, but many things are back online.
As far as lost posts go, here’s the report. I recovered the 176 published posts that were lost over the last 6 months by visiting my completely neglected Bloglines account. I saved off their feed output for my site and used a complicated Regex + WordPress import plugin to move all of the posts back into the database. Unfortunately, all comments on those posts were lost in the crash, and the one post (“Building A Shelf In WordPress”) that had more than one page now only contains the first page.
Asymptomatic wasn’t the only casualty of the crash. This server also hosted RedAlt and MicroWiki.net. MicroWiki was most affected by data loss, since the software is about 10 files in total and is hosted in Google’s Code repository. All of Berta’s data was lost, and all of my MicroWiki rhetoric was lost. Nothing that can’t be rebuilt over time though, and the site itself is already back with new data pouring in as I write this.
On the other hand, RedAlt is a big mess. There was a lot of custom code running on that server, especially that which implemented the specialized WordPress function reference. A lot of that was in a nearly-finished state in the backup that I have, but I have made little tweaks over the last six months that will be harder to recall.
Many plugin zip files are lost. I have backups, but if there were changes since then to plugins that were not in my Subversion repository (hosted on a separate server), then they’re not in those backups. It’s probably time to retire those plugins anyway.
Nonetheless, there are many things on RedAlt that are already back online. Kubrickr is back and better than before. You can now enter your Flickr username and see only your own CC-licensed photos for use as a header. The cropper is much improved as well.
It may take a little time, but I do intend on returning these services to at least their normal state, if not better.
As far as doing something to prevent this from happening in the future, I have taken steps to backup both the files and the MySQL data to a remote location every day. I’m using rsync to transfer the backup data to my lifetime StrongSpace account, which is big enough to hold the whole server. I have already made a snapshot of the server using the VPS tools provided by Unixshell, and so it should be a three-step process to restore the whole system, if need be. Restore the snapshot, rsync the files back to the server, execute the backup queries - done!
Hopefully, I won’t have to worry about that ever again.
