Journalspace Floats Belly-Up, Epic Database Failure Date: 2009-01-05 16:35:35
This would be pretty funny if thousands of people hadn't irreversibly lost their data. Web startups die all the time, so it's easy to miss the real juicy stuff. Usually, funding runs out, unanticipated legal problems develop or it's just that nobody cares about the service. But it wasn't any of these that killed Journalspace.
The incompetence of the Journalspace team, especially the 3rd-party admin who got fired ages ago for stealing and then sabotaging customer servers, is the material of mythic, truly epic, mind-blowing Fail that I'm sure people will talk about in allegory for a long time. For several hours at least. Here's what happened:
Journalspace was run off a single self-administrated server (OS X on a Mini apparently) and the team's single and only backup strategy was, get this, having two hard drives that continually mirror each other. No other backup or data security plan was in place. So there it sat, that lonely little Mini box, in some corner in some datacenter, with its two hard drives that always - by design - have exactly the same content. You can probably guess where this is going. One unresolved
SQL command later, everything was gone. Just like that. Nobody saw it coming, nobody cared it was bound to happen some day.
I have some pretty bad business experience to look back on. I had my fair share of funding problems, sleazy-ass legal bullshit, and staggering incompetence (including my own). But this takes the top of the list, easily. I even understand when people back up their DB only every few days, or even weeks. For some scenarios that may even be appropriate. But when you start hosting a service where thousands of users are continously contributing their own content, content that they trust into your hands only, this is fucking inexcusable.
I also don't understand this obsession with having your own dedicated server running somewhere. It's always more expensive than any other option. And those boxes are always in worse shape than the professional hosting services offered by providers. And you have so many better options to choose from. Need high dynamic load capabilities? Go to MediaTemple. Need your own grid of on-demand virtual servers? Use Amazon. Need high availability? Ask HP or Google. Need extreme security? Go to one of those military-rated datacenters in your neighborhood. Running a site with an Alexa rank of 106,881 on your poorly patched Mac Mini in some guy's basement? Unbelievably stupid and wasteful.
Now the domain name is for sale on eBay, no doubt scheduled to be acquired by shady some spam master looking for a quick gig to make some ad dollars. And that's it. But this enormous failure raises an even bigger issue. All of us web service users, we're putting our data all over the web. We're spending hours each day creating that stuff, and it all is spread among some thick-walled data bunkers that may or may not go down forever in the near future. The problem is, there is no way to actually own your data. I can't archive my Facebook messages anymore than I can reasonably save RSS items I liked from Google Reader, or Forum posts I enjoyed. You'd think that in a world where we're spending so much time churning out all of this personal crap, we'd actually associate a value with that content. But we don't. We just put it out there and nobody cares when it's gone. A friend of mine didn't update her Spaces blog for a few months, now it's all been deleted. Gone, no backup, no support. If our stuff has no value, then why even invest the time writing, photographing or filming it? But if it does have value, why isn't there some kind of personal data hub where you can take care of all your online stuff, at least for the purpose of backup and accumulation?
The incompetence of the Journalspace team, especially the 3rd-party admin who got fired ages ago for stealing and then sabotaging customer servers, is the material of mythic, truly epic, mind-blowing Fail that I'm sure people will talk about in allegory for a long time. For several hours at least. Here's what happened:
Journalspace was run off a single self-administrated server (OS X on a Mini apparently) and the team's single and only backup strategy was, get this, having two hard drives that continually mirror each other. No other backup or data security plan was in place. So there it sat, that lonely little Mini box, in some corner in some datacenter, with its two hard drives that always - by design - have exactly the same content. You can probably guess where this is going. One unresolved
DROP DATABASE journalspace;
SQL command later, everything was gone. Just like that. Nobody saw it coming, nobody cared it was bound to happen some day.
I have some pretty bad business experience to look back on. I had my fair share of funding problems, sleazy-ass legal bullshit, and staggering incompetence (including my own). But this takes the top of the list, easily. I even understand when people back up their DB only every few days, or even weeks. For some scenarios that may even be appropriate. But when you start hosting a service where thousands of users are continously contributing their own content, content that they trust into your hands only, this is fucking inexcusable.
I also don't understand this obsession with having your own dedicated server running somewhere. It's always more expensive than any other option. And those boxes are always in worse shape than the professional hosting services offered by providers. And you have so many better options to choose from. Need high dynamic load capabilities? Go to MediaTemple. Need your own grid of on-demand virtual servers? Use Amazon. Need high availability? Ask HP or Google. Need extreme security? Go to one of those military-rated datacenters in your neighborhood. Running a site with an Alexa rank of 106,881 on your poorly patched Mac Mini in some guy's basement? Unbelievably stupid and wasteful.
Now the domain name is for sale on eBay, no doubt scheduled to be acquired by shady some spam master looking for a quick gig to make some ad dollars. And that's it. But this enormous failure raises an even bigger issue. All of us web service users, we're putting our data all over the web. We're spending hours each day creating that stuff, and it all is spread among some thick-walled data bunkers that may or may not go down forever in the near future. The problem is, there is no way to actually own your data. I can't archive my Facebook messages anymore than I can reasonably save RSS items I liked from Google Reader, or Forum posts I enjoyed. You'd think that in a world where we're spending so much time churning out all of this personal crap, we'd actually associate a value with that content. But we don't. We just put it out there and nobody cares when it's gone. A friend of mine didn't update her Spaces blog for a few months, now it's all been deleted. Gone, no backup, no support. If our stuff has no value, then why even invest the time writing, photographing or filming it? But if it does have value, why isn't there some kind of personal data hub where you can take care of all your online stuff, at least for the purpose of backup and accumulation?
Comments
|
EMSPV says
(2009-01-07 06:13:00)
| |
 Vikki |
Vikki says
(2009-10-22 12:14:13)
I just lost a huge amount of a story I wrote with a friend on Facebook messages. All gone. |
It is quite interesting how the typical lazyness of an user to regularly backup his or her own data leads to a total loss of care regarding any kind of data. The act of *producing* new content seems to be easier and going much faster than actually *saving* (and not just publishing) this new content. Does that mean that we, the users who generate, do not care about our own stuff?