Posted: 8/21/2006 9:36:04 AM EDT
How long would you have a job if you took a customer facing web site down multiple times in one day?
|
|
CYA in intro to programming, that's funny I dunno, sites go down all the time, it happens, people expect it to happen (the reason that VOIP will never take over the phones, it doesn't work). you just work hard and get it back up. computers crash, it happens, they didn't pay for redundancy, they don't get 24/7/365 now, if you unplugged it, well, better start walking. 
|
|
Really depends on why the site went down. Stupid configuration mistakes or accidently tripped a breaker that the server is plugged into? There are varying degrees of error. Accidently clicking 'shut down' when remoted into a box is inexcusable....trippping over a wire in a poorly organized data center is a whole different issue. |
this one has learned well. pfffttt websites are small time in the grand scheme of the wonders we call IT. if your the web guy it's ALWAYS an application problem. if your the application guy it's ALWAYS a hardware problem. if your the hardware guy it's ALWAYS the application developers problem. If your the application developer then your ALWAYS busy no time to deal directly with the customer. you pass it off to the applications guy and the cycle repeats. 
|
You have to get creative when you wear *all* those hats. 
|
easily handled..... blame the vendor that is hardest to get in touch with. by the time they track them down on a wild goose chase the problem is resolved and you have had PLENTY of time to make the approprite cover story. remember... when doing the after action autopsy of the down time use BIG words, even invent a few if you have to. It makes management feel like you know exactly what is going on and confuses them. They won't ask questions becuase that would force them to admit they have no clue what you do. |
 You are wise in the ways of IT. |
i work for big blue..... 2/3 of my day is making excuses
|
|
Production or Test/Development? Test, we take them down all the time for various maintenance and updates, but the customers knows that can happen (and usually has requested the work being done.) If Production, you could probably get away with that once around here, get chastised, and move on. Twice you'd be in deep shit. Three times you'd likely be out the door. |
|
Well, I usually CYA, then turn it into a Sales Opportunity. "You know, this is a prime example of why you need a Hot Standby Disaster Recovery Site, which would also protect you in case of Hurricane, Terrorist Attack, or the 2nd Coming of the Flying Spaghetti Monster. We have some really cost-effective DR options that I could quote for you, if you're interested?" Works like a champ.  Futuristic |
Hell yes, I love this one. For me (I'm a Cisco IP Telephony phone geek/consulatant/engineer), it works something like this: "Your IPCC Express 3.5 server went down, which completely killed your call center. It's a shame that this happned, and the impact that it had on your business is substantial. But I've gotta tell ya - Cisco has listened to it's customers and has come up with a solution. With IPCC Express 4.x, you can have a hot standby server available that will fail over in less than 5 seconds. If you had this right now, the faulty server could have been fixed at your leisure. Now, I noticed that you've got a Smartnet SAU contract, so it won't cost you a dime for the software. I've got some availability next week, and I can take care of this for you for about $3500 in services. You'll have to buy another server for reduncancy, but that should only run you about $4000. So for $7500, we can make sure this never happens again. I bet that the downtime you experienced today cost you alot more than $7,500. the potential sales lost were no doubt at least triple that. What time do you have available next week for me to begin implementing the solution that's going to make you look better to your boss?" Tee hee. 
|
I might change that to something more along the lines of, ‘We can dramatically reduce the chances of this happening again.’ Just adding a little more CYA  edit for smiley |
Good point.
|
|
It would depend on which site, specifically how much money was lost and how much customer ill will was generated as a result. Where I work, there are about a half dozen that I wouild get seriously yelled at if I did something stupid that brought them down during business hours. There are also a dozen or so that nobody would be likely to notice. |
'tis true. Many a quota were exceeded on the basis of an irate customer being upsold because some dipshit tripped over a power cable. Inconsolably screaming CEOs want immediate rectification for their pain. Most times that level of service costs obscene amounts of money to provide. Our job as hardware and software vendors and developers is to have that solution on the shelf ready for deployment when those pure margin deals happen. Going from one "9" to the next is an exponential journey of the pocketbook. But that is why you pay for insurance, and good IT is just another form of risk management. |