posted
As many of you know, I work for Safeway. We're a large grocery chain, with something like 200,000 employees across the world (including Manila and India).
Anyways, I'm pretty sure that we have got to have one of the worst IT depts in the world...
Let me explain. When I came in at 4am, I was greeted by 15 callers in the queue. It's pretty unusual for us to be this busy at this time of day (we do have stores on the east coast), so I asked what was up.
Someone in IT apparently decided last night that our Active Directory system was working too well, and promptly screwed up all of the accounts in both of our domains (more than 20,000 accounts if you want to keep count). At this point, every caller will have to contact us to get their account unlocked/password reset so that they can get logged into the system. That is unless someone gets a really bright idea and resets all passwords to something new, which will cause everyone in the company to call in at one time.
I'm not looking forward for the corporate offices to come in this morning @ 6am (something like 10,000 people coming in at the same time).
Oh, we're supposedly in the holiday moratorium phase at this moment. We have been since Dec 1st. What this means is that if it was an official change, it was approved by one of our many VP's (we're so top heavy that it's not even funny).
This isn't the first time that someone's screwed with Active Directory and/or passwords therein this year. This would be the 4th time (possibly 5th, I've lost count), that they've done this this year.
Amazingly, the person who's done this every time so far still has a job with us. Apparently our company doesn't like to expunge people who cause large site outages (and quite possibly never punish them, either).
Current number of calls since the problem started at 6:45 last night: 650. John's facial expressions: priceless.
Our problems are compounded by a recent software upgrade to our OS2 servers that every store has that locks up printers. We've got a fix for it, but it takes 15 minutes to do (and you have to work solely on the problem). At the rate of software releases that we get, it'll be March before it's fixed.
Wee! I just got a call from our CIO (directly below the CEO). There's a phone outage affecting our Pleasanton offices. That means that it'll be a bit quieter when they get in, since they won't be able to call us (roughly 8,000 people). He's pretty pissed that Siemens says that they don't have to go out till later this morning. Good thing there are monkeys here that are higher ranking than me and they can get yelled at.
Well, I feel a bit better now that I've been able to rant (our ICQ server is down as well this morning, so there's no one to talk to).
Edit: there's more to this story, but I can't take anymore time away from the callers right now.
posted
So does this mean you're hiring competent people?
Or is that against corporate policy? (I'm assuming you got in due to the previously hired incompetent people doing the interviews; a glaring flaw in an otherwise devious plan)
Posts: 15770 | Registered: Dec 2001
| IP: Logged |
posted
<Thumbs through Bob's Giant Book of IT Strategies>
Ah, here it is!
quote: Whenever you are feeling vulnerable, not just in your employment but in any facet of your life, take steps to show how absolutely indispensable the IT staff are. Show how, seemingly through no fault of your own, the entire corporate communication system can be brought to a grinding halt. Outages affect the phones, e-mail, and every other method the drones can think of short of tin cans with string or smoke signals. Remember to always blame it on new software releases or the operating system in general. As you resolve the crisis (in your own sweet time), remember to mumble things like "man, I should just quit today and let them stew," or, "if this place ever actually mistreated me, imagine what I could do! And they'd never know!"
And remember our motto:
The data are OURS! And I'm not convinced you need it anyway!
posted
Actually, we've found that people who came into IT at Safeway from external companies are generally more competant than the people that have moved up from the stores (sorry Kenny and Shirley!). The problem is that there's a lot of people to go before we can get enough people in to replace the people from the stores.
In the meantime, our LAN group did a global password change and we've seen passed the fabled 40 call mark (our record is 99 calls in the queue, but I was out that day), so it's a busy day for me here.
Oh, I feel sorry for anyone who gets their prescriptions for our pharmacies.
Posts: 851 | Registered: Oct 2001
| IP: Logged |
posted
I wanted to make sure it wasn't just our engineers that had screwed it up.
One of the engineers is well known for being unable to spell most common words correctly (words like fatal, scales, etc). It's quite funny when you see three versions of the same summary (none of which work).
Another engineer is known for the amount of time that she spends on smoke breaks, decorating the call center, or complaining about how busy she is (must be tough to get paid that much).
If we had decent programmers, I'd love to see a ticketing system developed by us, but we can't even get a program to run the price updates to work correctly (one year now and running).
Posts: 851 | Registered: Oct 2001
| IP: Logged |
posted
We also developed our own ticketing system. And we have no official "programmers" on board. Just one of the guys of the help desk, frustrated with the system, and a "hobby programmer" on the side (but a VERY good one) developed our ticketing system. (how can anyone think of programming as a fun hobby -- I HATE programming.) It works really well, and now other companies have come over to take a look at it.
posted
*sigh* I can't take this. There's been 15 calls in the queue basically since I came in. Half the slack@sses around here don't feel the need to take any calls (two of them are actually putting up xmas decorations at this point!!).
1 hour 57 minutes to go today. I've taken 135 calls already, and I may be able to top my old record of 174 by the end of the day.
Tomorrow is going to be a mental health day, I just can't stand this place at all... Posts: 851 | Registered: Oct 2001
| IP: Logged |
Printer-friendly view of this topic
![[Wall Bash]](graemlins/wallbash.gif)
![[Big Grin]](biggrin.gif)
