quote:Our data center (Internap) lost all its power, including redundant backup power. We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.
[ January 14, 2005, 10:30 PM: Message edited by: BannaOj ]
Posted by Insanity Plea (Member # 2053) on :
Holy crap, that sounds like a sysadmin's nightmare...well at least startup of clustered servers (at least ones I've worked with) takes pressing about 30 keys. Though, theirs are probably doing more than the ones I played with did. Satyagraha
Posted by xnera (Member # 187) on :
Not the best thing to happen right after the acquisition by Six Apart that has all of LiveJournal in an uproar. The average user is not going to understand about things like colocation. They're just going to know their journal is down, and be upset about it. I feel for Brad, Lisa, et al right now.
Posted by xnera (Member # 187) on :
Oh, and just so you all know, you can always go to http://status.livejournal.org/ to check the status of LiveJournal. It's hosted on another site.
And I am bored as LiveJournal is down, so I will ramble about LiveJournal's backend.
I wonder how many clusters LiveJournal has now? I used to have a better sense of how many there were, but they've renamed a bunch of them and I've lost track. I know that there's Ribeye, FiletMignon, and Chef, but those are most definitely not all of them. Hmm.. Indigo might still be out there; I think it's the Inactive cluster now. Or did Indigo get renamed into FiletMignon? Am so confused.
I still miss the Santa cluster. Santa was always good to me. I'm on Ribeye now, I think.
They use a master-master setup, so each cluster has at least two machines, but there might be more. And I'm pretty sure they're all on InnoDB now. Chef was the last to get transferred over, and that was completed last weekend.
I think they still use a colocation site. I asked Brad if they're going to move the servers when the staff moves to the Six Apart offices, but he hasn't replied, alas.
--xnera, your resident LiveJournal geek
Posted by TomDavidson (Member # 124) on :
I was just thinking the other day that if you did a Venn diagram of slash fiction fans, people hosting webpages about the upcoming Firefly movie, cat owners, aspiring comic artists, and otaku, you'd wind up with an intersection in the middle consisting almost entirely of LiveJournal users.
Posted by BannaOj (Member # 3206) on :
xnera I was wondering if LiveJournal withdrawal was giving you the vapors
AJ
[ January 14, 2005, 10:31 PM: Message edited by: BannaOj ]
Posted by xnera (Member # 187) on :
Tom, I am so posting that to my journal when it comes back up!
AJ, not quite. But yeah, it feels weird not to be on LiveJournal right now. I was going to spend the night catching up on various things I'm behind on, and now I can't do that. Woe!
I am very, very tempted to hop into the #lj_support channel to see what's what, but I haven't done support in a few weeks so I don't feel like I really belong there. And they're probably just talking about music or food anyway.
Posted by Alcon (Member # 6645) on :
Wow... they get to manually reboot 100+ servers... sucks to be them!! Posted by BannaOj (Member # 3206) on :
you know I'd actually chat with you if you were on IM xnera
What is otaku anyway? Is it a genre of something? anime? I've heard the term but other than the infamous hatrack troll I didn't know it had significace.
AJ
Posted by xnera (Member # 187) on :
Otaku. It's a person, not a thing. Usually used in reference to anime fans. I grumbled at my TV when the Gravitation DVD extras said I was otaku for watching it in Japanese with English subtitles. Though to be honest I probably am otaku. *sighs*
Oh, by the way, it's correctly written as LiveJournal. Support volunteers get that drilled into our heads, so seeing it in all lowercase is making me twitchy.
xnera23 (9:12:59 PM): LiveJournal. LiveJournal. LiveJournal. That is how it should be. <a friend> (9:13:13 PM): Eljay xnera23 (9:13:29 PM): Eljay is my friend from Minnesota.
Posted by Synesthesia (Member # 4774) on :
Otaku is another word for nerds. In the US anime world, it's not a bad thing to be, in Japan you can be an otaku for anything, dolls, books, trees, you name it. It's looked down upon a bit...
Arg. I hate not having LJ on. But, at least I have my DJ which is better than LJ anyway. I get all twitchy when I can't write my random thoughts down. Especially since I saw INNOCENCE.
Posted by xnera (Member # 187) on :
Synth, would you have a DeadJournal code? I've been thinking of getting a DeadJournal to poke around on. Out of all the clone sites, I trust them the most.
I do NOT trust Mindsay. At all. Horrible site. Jury's still out on JournalFen and GreatestJournal.
I know otaku is becoming more positive now, but for a while I just knew it as negative, so I prefer not to think of myself as otaku, even if it fits.
Posted by Synesthesia (Member # 4774) on :
Sadly, I do not have a code... maybe when I get a paid account one day..
Posted by BannaOj (Member # 3206) on :
Less twitchy?
Hmm, I had no idea, though Steve's old roommate owned every episode of Ranma 1/2 ever released in the US or Japan.
AJ
Posted by Grisha (Member # 6871) on :
it says something slightly different when i try to go to live journal anyway:
quote: Our data center (Internap) lost all its power, including redundant backup power, for some unknown reason. (unknown to me, at least) We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.
basically the same though, it just says they don't know why this happened. Posted by xnera (Member # 187) on :
Thanks anyway, Synth. I'll probably end up paying the $5 to get a DeadJournal account. Eventually.
AJ: Posted by Synesthesia (Member # 4774) on :
Dude. He must have paid hundreds of dollars for that. That series is a black hole, never ending!
Posted by xnera (Member # 187) on :
I have spent nearly $200 on manga in the past eight months. *winces* But dude, it's good! Right now I am hooked on Fruits Basket, because it is easy to fine in the bookstore. Still longing for Mars #6.
Posted by xnera (Member # 187) on :
So, any thoughts on the Six Apart/LiveJournal dealie? Personally, I think this could be great technologically, though I worry about the effect it will have on the culture of the site.
Posted by xnera (Member # 187) on :
quote:Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable), and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.
Posted by BannaOj (Member # 3206) on :
Syn yes he did. Unfortunately as a consequence he was a bit unreliable on the rent, including dissappearing for a week to an Anime Convention when the rent was due, forcing Steve to move out.
Steve said he didn't have quite *everything* but did have at least 2 or 3 seasons of Japaneese-only in advance of when they were released to the US. He'd print out translations to english so that he could follow along.
AJ
Posted by J T Stryker (Member # 6300) on :
I have no idea how I'll be able to sleep without posting my daily rant about how I'm bored and miss my GF on LJ....
Posted by Trisha the Severe Hottie (Member # 6000) on :
I'm into LiveJournal but I don't do/have/be any of those things. But then, I only have two people on my friends list and my main reader isn't on LiveJournal.
Posted by Synesthesia (Member # 4774) on :
I just need to write essay fragments... My DJ is secret, so only one person reads that, but anyone can read my LJ For a person afraid of being read, I sure do like an audience.
Posted by TomDavidson (Member # 124) on :
"For a person afraid of being read, I sure do like an audience."
I think you just described the primary appeal of LiveJournal. There are thousands of possibly bisexual introverts around the world desperately hoping that someone will read something that they wouldn't share with their friends. Posted by Synesthesia (Member # 4774) on :
Maybe Or perhaps I just need my LJ and DJ for practice.
Posted by Trisha the Severe Hottie (Member # 6000) on :
Practice?
Posted by Synesthesia (Member # 4774) on :
Yes... I want to become a writer... But, all of my writing is random, nonsensical stream of conscience observations.
Posted by J T Stryker (Member # 6300) on :
*shakes in corner*
LJ withdrawal is almost as bad as Hatrack withdrawal.
Posted by Anna (Member # 2582) on :
I just began to use it and it crashes ! Posted by xnera (Member # 187) on :
quote:Update #2, 10:11 pm: So far so good. Things are checking out, but we're being paranoid. A few annoying issues, but nothing that's not fixable. We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again. In the past we've always trusted Internap's insanely redundant power and UPS systems, but now that this has happened to us twice, we realize the first time wasn't a total freak coincidence. C'est la vie.
Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn't commit to disk when asked, so InnoDB's durability wasn't so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We'll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we're going to take a bit of a break while the binlogs replay... Again, our apologies for the downtime. This has definitely been an experience.
Update #4: 9:12 am: We're back at it. We'll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.
Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh* We're still at it. We'll be doing a full write-up when we're done, including what we'll be changing to make sure verify/restore operations don't take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn't fast enough. And also that some of our machine's storage subsystems lie. Anyway, we're still at it... it's long because we're making sure to back up even the partially out of sync databases that we're restoring, just in case we encounter any problems down the road with the restored copy, we'll be able to merge them. And unfortunately backups and networks are too slow.
Not only has LiveJournal been done for a day, but my AIM bot has been down as well. I just can't get him to connect. Am very right now.
Posted by blacwolve (Member # 2972) on :
LJ withdrawal is far worse than Hatrack withdrawal. Now I can't read Nocturne_Alley either, which is driving me insane!
Posted by Theca (Member # 1629) on :
You keep talking about AIM bot. What is it?
Posted by breyerchic04 (Member # 6423) on :
I need lj too, though have no reason to post anything, just it feels like I haven't talked to my friends in forever without it for a day.
Posted by Katie DeShane (Member # 7026) on :
damn.. and i just paid for a year's worth of full access LJ...
am stuck here in the barracks watching Jack.. blahhh (who knew J.Lo was in it???)
someone want to entertain me and lament with me about Eljay being down??
talk to me peoples.. I hate how Hatrack doesn't have a reply-notification like LJ or some of the other places I'm a part of have.
*grumbles off into the boring night*
~katie~
katie23jazz - AIM | irishtechno21 - yahell
Posted by Narnia (Member # 1071) on :
mmmm. three day weekend. Ice storm. Stuck in my house.
So I have TIME to update my journal. Drat. I miss it.
Posted by reader (Member # 3888) on :
Livejournal is back up.
It's slow - probably because of all the people checking in, and not all accounts seem to be back up, but for the most part, it's back.
Posted by Narnia (Member # 1071) on :
Oh. Really? I haven't gotten it to load. But I haven't gotten the white screen of death either, so I believe you.
Posted by reader (Member # 3888) on :
I have a very fast internet connection, so that may be a factor... Try again soon. Posted by Synesthesia (Member # 4774) on :
It's back?!?! I really miss LJ and spouting out play by play scenes from movies... But, I can do that on DJ too
Still not back
[ January 15, 2005, 09:42 PM: Message edited by: Synesthesia ]
Posted by Katie DeShane (Member # 7026) on :
LJ is up!!!
YAYness!!!!!!
okies I have a super ass slloooow laptop so yeah it's def up now!
Posted by xnera (Member # 187) on :
Yay for LiveJournal being back up!
quote: You keep talking about AIM bot. What is it?
It's a chatterbot. Basically, it's a program that replies when you talk to it on AIM. I didn't write it from scratch, but I spent many, many hours tweaking the source code, so it's been a big part of my life the past seven months.
And now it's not working. I am crying here.
Posted by JaneX (Member # 2026) on :
Mine is still down.
~Jane~
Posted by Narnia (Member # 1071) on :
My account still isn't working...and it's not my connection (I have broad band). Maybe tomorrow.
Posted by xnera (Member # 187) on :
They're still having problems with some of the clusters.
quote: We have all database clusters except one back online at this time. Unfortunately MadCow is still recovering, and we expect it to be back online by around 10 PM PST tomorrow. (Where am I?).
Posted by quidscribis (Member # 5124) on :
You have to pay for LiveJournal? Why would you do that when you can blog for free (or, at least, much cheaper)?
Posted by JaneX (Member # 2026) on :
You don't have to pay. A paid account just gives you more options for customization and little bonuses like that.
Mine is back up now!
~Jane~
Posted by blacwolve (Member # 2972) on :
what's a cluster?
Posted by reader (Member # 3888) on :
I'm on MadCow - or at least, I was last time I checked, which was just a couple of weeks ago - and I was back up right away. But maybe my journal is just a fluke....
Edit: I just checked again, and I've been switched to TBone, so that explains it.
[ January 16, 2005, 03:36 AM: Message edited by: reader ]
Posted by Noemon (Member # 1115) on :
![[Wink]](wink.gif)
![[Smile]](smile.gif)
But yeah, it feels weird not to be on LiveJournal right now. I was going to spend the night catching up on various things I'm behind on, and now I can't do that. Woe!![[ROFL]](graemlins/rofl.gif)
![[Dont Know]](graemlins/dontknow.gif)
![[Kiss]](graemlins/kiss.gif)
![[Mad]](mad.gif)
![[Wall Bash]](graemlins/wallbash.gif)
![[Cry]](graemlins/cry.gif)
![[Big Grin]](biggrin.gif)
![[Frown]](frown.gif)
![[Party]](graemlins/party.gif)
![[The Wave]](graemlins/thewave.gif)