on the nature of fuckups. or something.

[rmd]

apparently, i was due for a good solid shoot-yourself-in-the-foot fuckup.

words to the wise from the voice of experience: if you have two devices working as a primary/failover pair, and you want to test them both, you have to test through each of them and not twice through one of them. *facepalm*

*sigh* this is the sort of thing that would've had me wearing the 'bonehead' headpiece when i was at ftp.

on the other hand...

back very shortly after i started at msft, working for the then-fledgling msn, i had an coworker for about 2 months. the guy started, and very shortly after he took another job wrangling networks at some HMO or something. i was told after he left that he apparently thought the stress of working at msft was more than he wanted to deal with.

and i couldn't quite understand this. because at msft, OH MY GOD PEOPLE MIGHT NOT BE ABLE TO READ THEIR EMAIL if there was a fuckup. or, you know, have a fuckup show up above the fold on newspapers all around the world. (yeah, that happened. not my fault, although i did kick myself for not diagnosing and solving it before other people did.) but at an HMO, people might ACTUALLY FUCKING DIE due to information technology fuckups.

Comments

[dpolicar]

(nods) Back when I cared, the mantra "It's only a billing system; nobody dies" was oft-repeated,

[feste-sylvain.livejournal.com]

At PTC, I did bug triage, where a "Critical" bug was one that either crashed the CAD/CAM program or lost data.

And this classification worked quite well at my next job, which did automatic Linux package maintenance.

But at the job after that, which controlled 25-ton factory-floor robots, a "Critical" bug was one which could kill or maim a worker.

And that was the outfit which didn't have any QA.

In my current job, our 'bots don't exactly kill people, but they can expose users to enemy fire if something goes wrong, so that's now the definition of "Critical".

[rmd.livejournal.com]

yeah, see, those are critical things that are REALLY ACTUALLY CRITICAL.

given how stressed i sometimes get over things like email and tcp/ip, i worry about how i'd handle the stress of actual lives in the balance.

i assume i'd deal. because humans are flexible that way. i'd probably drink more, though. or maybe i'd drink less...

[solarbird]

You should've seen the Excel team freak when they discovered some hospitals were using it for critical medical care. Oh man.

[cz_unit]

Yeah, it helps to test redundant things, every once in awhile I'll pull a cable just to see if the servers fail over properly. Best to do it when there are people there.

Of course sometimes you're fucked no matter what: Having two separate T3 connections coming into your data room with NIDs located 10 blocks in opposite directions will protect you in a 5 block failure, but not in a 30 block failure. Sometimes shit just happens.

As for the HMO, well it's probably something people pay for so they usually can deal with the system being down. Free shit of course has to be up at any time because that's a God-Given *ENTITLEMENT* and we all know that the people in china/india/wherever will do anything possible to keep systems running 7*25*forever so you better do it too.

Ahem. These days, I'm pleased if anything works.

CZ

[cz_unit]

*nod* I have no problem working with life-critical systems, you just use tools designed for the purpose. You don't use Linux, you don't use Windows 2003/08/whatever. You use systems more like the old Tandem boxes or Sequent.

The real trouble is when people demand 24*7 using the latest Windows frob app that is loaded with bugs. There's nothing quite as funny as telling the Windows people we need to reboot their app server because well... Windows has yet another buffer bug in the assinine 3Com 3+OPen network stack.

CZ

Not as critical as my boss thinks

[kellyjmf.livejournal.com]

I while back I tried to tell our telecom chick that she had made a mistake and my BlackBerry wasn't supposed to be a phone as well. She said that it need to be a phone in case they needed to reach me in an emergency.

I tried to imagine an emergency that would require me to fix a JDE report in the middle of the night or an IT problem that couldn't be better solved by any one of the twenty other people in our department. But hey, if I'm the only one left, we have a more serious problem that a phone call is not going to remedy.

But free cell phone, yay!

At a conference, a vendor was telling me all about the joys of RFID tags for warehouse inventory. I told him our assets were commercial retail spaces that tended to stay where you put them. And if they start moving or you can't find them, there are usually hurricanes involved, or Godzilla.