DriveSavers called today to inform me that the data was unrecoverable.
Here is what happened: the server which held the journalspace data had two large drives in a RAID configuration. As data is written (such as saving an item to the database), it's automatically copied to both drives, as a backup mechanism.
The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that's gone bad, and the server is back to operating with two redundant drives.
But that's not what happened here. There was no hardware failure. Both drives are operating fine; DriveSavers had no problem in making images of the drives. The data was simply gone. Overwritten.
The data server had only one purpose: maintaining the journalspace database. There were no other web sites or processes running on the server, and it would be impossible for a software bug in journalspace to overwrite the drives, sector by sector.
The list of potential causes for this disaster is a short one. It includes a catastrophic failure by the operating system (OS X Server, in case you're interested), or a deliberate effort. A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can't rule out the possibility of additional sabotage.
But, clearly, we failed to take the steps to prevent this from happening. And for that we are very sorry.
So, after nearly six years, journalspace is no more.
If you haven't yet, visit Dorrie's Fun Forum; it's operated by a long-time journalspace member. If you're continuing your blog elsewhere, you can post the URL there so people can keep up with you.
We're considering releasing the journalspace source code to the open source community. We may also sell the journalspace domain and trademarks. Follow us on twitter at twitter.com/jsupgrades for news.
It's a shame, but then most everyone I know who used JS switched over to LJ years ago.
Out of curiousity, if the cause was in fact sabotage what legal action can the JS team take?
--j_k
[ October 10, 2009, 11:46 PM: Message edited by: James Tiberius Kirk ]
Posts: 3617 | Registered: Dec 2001
| IP: Logged |
posted
If they had any operating revenue, they can sue for pretty enormous damages. And it's a felony, so they'd get a fair bit of retribution, too.
Posts: 37449 | Registered: May 1999
| IP: Logged |
posted
Yeah, though most likely anyone involved would have no assets worth anything.
This, people, is what happens when you lack/don't listen to competent IT administrators and have no backup plan. RAID is not a backup plan, RAID is a way of reducing failures that require restores from backups.
Posts: 15770 | Registered: Dec 2001
| IP: Logged |
posted
Amen. The idea that they had never taken an off-site backup in years of operation is mind-boggling to me. Not every service needs a daily or weekly backup -- although I'd think a journaling service probably does, to be honest with you -- but a monthly one seems like a bare minimum.
Posts: 37449 | Registered: May 1999
| IP: Logged |
posted
I imagine that this news item has already served as a catalyst for some organizations to reconsider their backup plans. At my workplace, we have regular backups, depending on the systems. They range from hourly to weekly, and we also do an off-site backup once a month.
We nearly had a nightmare the week before Christmas. I work in the IT office at a community college. In the main building, where I work, the HVAC system is antiquated and consists of a patchwork of various pieces. At our end of the building, the system terminates. In laymen's terms, it's at the bottom of the circuit, which means it collects all the sediment from the rest of the system and slowly builds up until a break occurs. The ventilation units in the ceilings, where the pipes are run, is always the culprit. Our server rack sits under a patchwork of pipes that would make Mario and Luigi proud. There have been four breaks in adjacent offices in the past year.
It finally happened, as it was bound to do. A water pipe in the heater unit (in the ceiling) burst, and it showered a lovely, rusty-colored hot water onto the floor. This break was no more than a foot away from our network core, and it was only six feet away from our server rack. We know we dodged a bullet; at least, I'm comfortable saying that all my co-workers agree with that assessment. I'm not so sure our administrators can comprehend the situation, though.
Posts: 1813 | Registered: Apr 2001
| IP: Logged |
posted
Can you imagine? The fall of myspace would be akin to the fall of the Berlin Wall. Hasselhoff could hold a concert, the CIA could have NO IDEA it was about to happen.
A million people would run outside and join hands and sing: "We're not gonna take it! No! We're not gonna take it!"
Posts: 9912 | Registered: Nov 2005
| IP: Logged |
posted
Heh...having worked with a couple of OS X servers, I'm leaning more to catastrophic failure (software RAID in 10.4 is...awful. I have horror stories. Actually, software RAID in general is awful.)
As a side note, the company I work for does network infrastructure support for Drivesavers. They're good people.
Posts: 3003 | Registered: Oct 2004
| IP: Logged |
posted
JTK - it could have simply been human error at Journalspace, as well, and no one wants to own up to it (so much easier to blame sabotage).
We had a situation once with one server having issues - and this server was automatically mirrored to the disaster recovery server site. The plan was the wipe the main server (install new hardware, actually, I believe) - and then copy the data from the mirrored site back over to the production side. However, someone forgot to turn off the auto copy, and when the copy-over kicked on, it copied the now-blank drives from the production side over to the DR site, so then both servers were blank. (Yeah, luckily we had tons of backups on tape, etc. Just took us a long time to get it restored properly).
I may not be remembering that exactly, as I wasn't part of the team that was involved in that, but it sure was the buzz around here for awhile. Human error does happen.
Posts: 9538 | Registered: Aug 2003
| IP: Logged |
quote:Well, this is shaping up to be one of the biggest disasters in the history of cloud computing, and certainly the largest blow to Danger and the Sidekick platform: T-Mobile's now reporting that personal data stored on Sidekicks has "almost certainly has been lost as a result of a server failure at Microsoft/Danger." They're still looking for a way to recover it, but they're not giving users a lot of hope -- meanwhile, servers are still on the fritz and customers are being advised not to let their devices power down because anything that's still on there will be lost the next time the device is turned on. Another communique is promised from T-Mobile on Monday to give everyone a status update on the recovery efforts, but at this point, it's not looking good at all.
posted
We had major computer issues at work tonight (Saturday). Our IT guy had to come in on his night off and work for 4-5 hours to bring everything back up. He replaced two ethernet switches. There were half a dozen people sitting around for part of this, because without access to the servers, we can't work on our pages (newspaper), can't write stories, can't tone photos, can't process obits, can't access the Internet. It was scary.
We discussed the possibility of going 20 miles to the next town where another paper in our chain is located (and our printing press) - but then we would have to backup the pages, ads, etc. and transport them. We don't have any system in place for this.
IT guy is planning to check in at work on Sunday. I'm thinking I will, too. Our managing editor is going to be training a new person in layout and I don't think it would hurt to have an extra person in case something else goes wrong.
Posts: 2034 | Registered: Apr 2004
| IP: Logged |
Printer-friendly view of this topic
