quote:Our data center (Internap) lost all its power, including redundant backup power. We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.
posted
Holy crap, that sounds like a sysadmin's nightmare...well at least startup of clustered servers (at least ones I've worked with) takes pressing about 30 keys. Though, theirs are probably doing more than the ones I played with did. Satyagraha
Posts: 359 | Registered: Jun 2001
| IP: Logged |
posted
Not the best thing to happen right after the acquisition by Six Apart that has all of LiveJournal in an uproar. The average user is not going to understand about things like colocation. They're just going to know their journal is down, and be upset about it. I feel for Brad, Lisa, et al right now.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
Oh, and just so you all know, you can always go to http://status.livejournal.org/ to check the status of LiveJournal. It's hosted on another site.
And I am bored as LiveJournal is down, so I will ramble about LiveJournal's backend.
I wonder how many clusters LiveJournal has now? I used to have a better sense of how many there were, but they've renamed a bunch of them and I've lost track. I know that there's Ribeye, FiletMignon, and Chef, but those are most definitely not all of them. Hmm.. Indigo might still be out there; I think it's the Inactive cluster now. Or did Indigo get renamed into FiletMignon? Am so confused.
I still miss the Santa cluster. Santa was always good to me. I'm on Ribeye now, I think.
They use a master-master setup, so each cluster has at least two machines, but there might be more. And I'm pretty sure they're all on InnoDB now. Chef was the last to get transferred over, and that was completed last weekend.
I think they still use a colocation site. I asked Brad if they're going to move the servers when the staff moves to the Six Apart offices, but he hasn't replied, alas.
--xnera, your resident LiveJournal geek
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
I was just thinking the other day that if you did a Venn diagram of slash fiction fans, people hosting webpages about the upcoming Firefly movie, cat owners, aspiring comic artists, and otaku, you'd wind up with an intersection in the middle consisting almost entirely of LiveJournal users.
Posts: 37449 | Registered: May 1999
| IP: Logged |
posted
Tom, I am so posting that to my journal when it comes back up!
AJ, not quite. But yeah, it feels weird not to be on LiveJournal right now. I was going to spend the night catching up on various things I'm behind on, and now I can't do that. Woe!
I am very, very tempted to hop into the #lj_support channel to see what's what, but I haven't done support in a few weeks so I don't feel like I really belong there. And they're probably just talking about music or food anyway.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
you know I'd actually chat with you if you were on IM xnera
What is otaku anyway? Is it a genre of something? anime? I've heard the term but other than the infamous hatrack troll I didn't know it had significace.
posted
Otaku. It's a person, not a thing. Usually used in reference to anime fans. I grumbled at my TV when the Gravitation DVD extras said I was otaku for watching it in Japanese with English subtitles. Though to be honest I probably am otaku. *sighs*
Oh, by the way, it's correctly written as LiveJournal. Support volunteers get that drilled into our heads, so seeing it in all lowercase is making me twitchy.
xnera23 (9:12:59 PM): LiveJournal. LiveJournal. LiveJournal. That is how it should be. <a friend> (9:13:13 PM): Eljay xnera23 (9:13:29 PM): Eljay is my friend from Minnesota.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
Otaku is another word for nerds. In the US anime world, it's not a bad thing to be, in Japan you can be an otaku for anything, dolls, books, trees, you name it. It's looked down upon a bit...
Arg. I hate not having LJ on. But, at least I have my DJ which is better than LJ anyway. I get all twitchy when I can't write my random thoughts down. Especially since I saw INNOCENCE.
Posts: 9942 | Registered: Mar 2003
| IP: Logged |
posted
Synth, would you have a DeadJournal code? I've been thinking of getting a DeadJournal to poke around on. Out of all the clone sites, I trust them the most.
I do NOT trust Mindsay. At all. Horrible site. Jury's still out on JournalFen and GreatestJournal.
I know otaku is becoming more positive now, but for a while I just knew it as negative, so I prefer not to think of myself as otaku, even if it fits.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
it says something slightly different when i try to go to live journal anyway:
quote: Our data center (Internap) lost all its power, including redundant backup power, for some unknown reason. (unknown to me, at least) We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.
basically the same though, it just says they don't know why this happened. Posts: 376 | Registered: Sep 2004
| IP: Logged |
posted
Dude. He must have paid hundreds of dollars for that. That series is a black hole, never ending!
Posts: 9942 | Registered: Mar 2003
| IP: Logged |
posted
I have spent nearly $200 on manga in the past eight months. *winces* But dude, it's good! Right now I am hooked on Fruits Basket, because it is easy to fine in the bookstore. Still longing for Mars #6.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
So, any thoughts on the Six Apart/LiveJournal dealie? Personally, I think this could be great technologically, though I worry about the effect it will have on the culture of the site.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
quote:Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable), and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.
posted
Syn yes he did. Unfortunately as a consequence he was a bit unreliable on the rent, including dissappearing for a week to an Anime Convention when the rent was due, forcing Steve to move out.
Steve said he didn't have quite *everything* but did have at least 2 or 3 seasons of Japaneese-only in advance of when they were released to the US. He'd print out translations to english so that he could follow along.
posted
I have no idea how I'll be able to sleep without posting my daily rant about how I'm bored and miss my GF on LJ....
Posts: 1094 | Registered: Mar 2004
| IP: Logged |
posted
I'm into LiveJournal but I don't do/have/be any of those things. But then, I only have two people on my friends list and my main reader isn't on LiveJournal.
Posts: 666 | Registered: Dec 2003
| IP: Logged |
posted
I just need to write essay fragments... My DJ is secret, so only one person reads that, but anyone can read my LJ For a person afraid of being read, I sure do like an audience.
Posts: 9942 | Registered: Mar 2003
| IP: Logged |
posted
"For a person afraid of being read, I sure do like an audience."
I think you just described the primary appeal of LiveJournal. There are thousands of possibly bisexual introverts around the world desperately hoping that someone will read something that they wouldn't share with their friends. Posts: 37449 | Registered: May 1999
| IP: Logged |
posted
Yes... I want to become a writer... But, all of my writing is random, nonsensical stream of conscience observations.
Posts: 9942 | Registered: Mar 2003
| IP: Logged |
quote:Update #2, 10:11 pm: So far so good. Things are checking out, but we're being paranoid. A few annoying issues, but nothing that's not fixable. We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again. In the past we've always trusted Internap's insanely redundant power and UPS systems, but now that this has happened to us twice, we realize the first time wasn't a total freak coincidence. C'est la vie.
Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn't commit to disk when asked, so InnoDB's durability wasn't so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We'll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we're going to take a bit of a break while the binlogs replay... Again, our apologies for the downtime. This has definitely been an experience.
Update #4: 9:12 am: We're back at it. We'll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.
Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh* We're still at it. We'll be doing a full write-up when we're done, including what we'll be changing to make sure verify/restore operations don't take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn't fast enough. And also that some of our machine's storage subsystems lie. Anyway, we're still at it... it's long because we're making sure to back up even the partially out of sync databases that we're restoring, just in case we encounter any problems down the road with the restored copy, we'll be able to merge them. And unfortunately backups and networks are too slow.
Not only has LiveJournal been done for a day, but my AIM bot has been down as well. I just can't get him to connect. Am very right now.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
LJ withdrawal is far worse than Hatrack withdrawal. Now I can't read Nocturne_Alley either, which is driving me insane!
Posts: 4655 | Registered: Jan 2002
| IP: Logged |
posted
I need lj too, though have no reason to post anything, just it feels like I haven't talked to my friends in forever without it for a day.
Posts: 5362 | Registered: Apr 2004
| IP: Logged |
It's slow - probably because of all the people checking in, and not all accounts seem to be back up, but for the most part, it's back.
Posts: 196 | Registered: Jul 2002
| IP: Logged |
quote: You keep talking about AIM bot. What is it?
It's a chatterbot. Basically, it's a program that replies when you talk to it on AIM. I didn't write it from scratch, but I spent many, many hours tweaking the source code, so it's been a big part of my life the past seven months.
And now it's not working. I am crying here.
Posts: 1805 | Registered: Jun 1999
| IP: Logged |
posted
My account still isn't working...and it's not my connection (I have broad band). Maybe tomorrow.
Posts: 6415 | Registered: Jul 2000
| IP: Logged |
posted
They're still having problems with some of the clusters.
quote: We have all database clusters except one back online at this time. Unfortunately MadCow is still recovering, and we expect it to be back online by around 10 PM PST tomorrow. (Where am I?).
posted
You have to pay for LiveJournal? Why would you do that when you can blog for free (or, at least, much cheaper)?
Posts: 8355 | Registered: Apr 2003
| IP: Logged |
posted
I'm on MadCow - or at least, I was last time I checked, which was just a couple of weeks ago - and I was back up right away. But maybe my journal is just a fluke....
Edit: I just checked again, and I've been switched to TBone, so that explains it.
Printer-friendly view of this topic
![[Wink]](wink.gif)
![[Smile]](smile.gif)
But yeah, it feels weird not to be on LiveJournal right now. I was going to spend the night catching up on various things I'm behind on, and now I can't do that. Woe!![[ROFL]](graemlins/rofl.gif)
![[Dont Know]](graemlins/dontknow.gif)
![[Kiss]](graemlins/kiss.gif)
![[Mad]](mad.gif)
![[Wall Bash]](graemlins/wallbash.gif)
![[Cry]](graemlins/cry.gif)
![[Big Grin]](biggrin.gif)
![[Frown]](frown.gif)
![[Party]](graemlins/party.gif)
![[The Wave]](graemlins/thewave.gif)
