| View previous topic :: View next topic |
| Author |
Message |
UKVIKING Senior Itiler

Joined: Sep 16, 2006 Posts: 3110 Location: London, UK
|
Posted: Thu Feb 14, 2008 7:12 pm Post subject: Change Management Horro Stories |
|
|
Folks,
Any thoughts about have a list of the Horror stories about poorly planned / implemented changes
This will help any one use them to provide ammo for having good change mgmt
News stories but no link
The latest one is Blackberry upgrade in US ... giggle _________________ John Hardesty
ITSM Manager's Certificate (Red Badge)
Change Management is POWER & CONTROL. /....evil laughter |
|
| Back to top |
|
 |
Ed Senior Itiler

Joined: Feb 28, 2006 Posts: 411 Location: Coventry, England
|
Posted: Thu Feb 14, 2008 9:18 pm Post subject: |
|
|
What about the stories that came from the British Government putting in a brand new system, and it cost double the estimate and didn't work. This was in the mid 80s. This is why we now have ITIL!!!!!
Also look at the problems for the Passport service, fairly recently, when they put in a new system. This gave all sorts of problems
Regards
Ed |
|
| Back to top |
|
 |
Ed Senior Itiler

Joined: Feb 28, 2006 Posts: 411 Location: Coventry, England
|
Posted: Thu Feb 14, 2008 9:23 pm Post subject: |
|
|
Another story was the Inland Revenue cock-up in 2000 (See BBC news 01/03/2000) Passport foul up was in 1999, heaps of others
Regards
Ed |
|
| Back to top |
|
 |
mg10_86 Itiler

Joined: Jan 21, 2008 Posts: 21 Location: Canada
|
Posted: Fri Feb 15, 2008 12:52 am Post subject: |
|
|
Canada's gun registry system.
Estimated cost $119 million. Actual cost $2 billion.
This is in Canadian currency, which is now in parity with the US dollar. |
|
| Back to top |
|
 |
Ines Newbie


Joined: Mar 21, 2006 Posts: 18
|
Posted: Tue Feb 26, 2008 1:35 pm Post subject: |
|
|
| Then there is Melbourne's (Austalia) "MYKI" public transport smartcard system, looking to be implemented at least 2 years late, and "..The system will cost about $1 billion, not the $494 million figure the Government has been using..." |
|
| Back to top |
|
 |
Skinnera Senior Itiler

Joined: May 07, 2005 Posts: 121 Location: UK
|
Posted: Tue Feb 26, 2008 7:25 pm Post subject: |
|
|
So I don't think any of these are Change (RFC) horror stories... these are just examples of badly run projects aren't they...?
I've been expecting to see things like allowing 2 Major Changes to go ahead on subsequent days to the systems that were central to an organisation's online and retail stores respectively and them both going badly wrong, and only at that stage it becoming obvious that the backout plans were badly flawed and interdependant...
If that had really happened ( ) the effect would have been to cripple both sales channels for about 4 days and it came about because the compound risk wasn't assessed properly - only the individual risk of each Change activity.
I'd further imagine that it would have lead to a significant bollocking for the Change team (and Head of...) swiftly followed by an equally significant revamp of how Major Changes were assessed and signed off...
Last edited by Skinnera on Tue Feb 26, 2008 8:33 pm; edited 1 time in total |
|
| Back to top |
|
 |
UrgentJensen Senior Itiler

Joined: Feb 23, 2005 Posts: 458 Location: London
|
Posted: Tue Feb 26, 2008 8:11 pm Post subject: |
|
|
Morning,
In my experience the worst CM related stories are when a major change is planned and looking good, but shortly afterwards it becomes apparent that all is not well.
For example, once we were having a data centre power down and whilst that was going ahead a couple of the network guys thought it would be the ideal time to upgrade the config on the core switch. Trouble is that switch was single point of failure and the entire company completley disappeared off the radar for a while.
Oh, and this is my favorite: Due to pressures a customer on a poorly run project I found that one of the techs had decided he couldn't wait for slot to be allocated in the server room for his new box and we found out a while later that he'd put it live... lying on the floor in a closet. With four other servers stacked on top. F&*king unbelievable.
Oh the trials and tribulations...
UJ _________________ Did I just say that out loud?
(Beige badge) |
|
| Back to top |
|
 |
Timo Senior Itiler

Joined: Oct 26, 2007 Posts: 295 Location: Calgary, Canada
|
Posted: Wed Feb 27, 2008 7:21 am Post subject: |
|
|
How about going from ITIL v2 to v3?  |
|
| Back to top |
|
 |
UKVIKING Senior Itiler

Joined: Sep 16, 2006 Posts: 3110 Location: London, UK
|
Posted: Wed Feb 27, 2008 7:27 am Post subject: |
|
|
ahhhhhhhhhhhhhhhhhhhhhhhhh ahhhhhhhhhhhhhhh
running screaming from the room _________________ John Hardesty
ITSM Manager's Certificate (Red Badge)
Change Management is POWER & CONTROL. /....evil laughter |
|
| Back to top |
|
 |
elewis33 Newbie


Joined: Apr 01, 2008 Posts: 12 Location: Salt Lake City, UT USA
|
Posted: Wed Apr 02, 2008 5:15 am Post subject: |
|
|
| Skinnera wrote: | | So I don't think any of these are Change (RFC) horror stories... these are just examples of badly run projects aren't they...? |
I'd have to agree that the first few replies were more about bad project management. This topic is near and dear to me as I'm struggling with our change manager over that very question.
You can view it here if you have any thoughts on the topic: http://www.itilcommunity.com/modules.php?name=Forums&file=viewtopic&t=3347
Earl |
|
| Back to top |
|
 |
OhioScott Itiler

Joined: Oct 29, 2007 Posts: 26
|
Posted: Tue Apr 15, 2008 4:55 am Post subject: |
|
|
New CIO forced through a mainframe patch from his former employer (IBM) over the objections of not only the Change Manager but also the Mainframe Support Team because the patch had not been tested.
Patch brought down the mainframe... luckily it was inside the maintenance window on a Saturday night so they were able to recover. CIO then forces through a quick and dirty fix from IBM to the patch on Sunday afternoon again over the objections of all involved. Result was that the mainframe is knocked out for 3 business days for a Fortune 100 insurance company.
Blame fell on the IT organization as a whole for failing to meet our customer's availability service levels while the CIO was not held accountable for any of the error. He used the situation to highlight inadequacies within our disaster recovery processes to gain a bonus from the board of directors! |
|
| Back to top |
|
 |
Doober Itiler

Joined: Apr 04, 2008 Posts: 30
|
Posted: Tue Apr 15, 2008 6:38 am Post subject: |
|
|
| OhioScott wrote: | | Blame fell on the IT organization as a whole for failing to meet our customer's availability service levels while the CIO was not held accountable for any of the error. He used the situation to highlight inadequacies within our disaster recovery processes to gain a bonus from the board of directors! |
What an insufferable prick! Our CIO has been a champion of every process improvement our organization has seen for the last 5 years. There's nothing in our organization (process wise) that he hasn't lead by example. Lamentably, he's leaving our organization in a few weeks to try a new industry. I pray we get someone even half as competant, and not one of these Scourge-Of-Reason dingbats you guys landed under.  |
|
| Back to top |
|
 |
DYbeach Senior Itiler

Joined: May 25, 2008 Posts: 413 Location: Sydney, Australia
|
Posted: Fri Jun 20, 2008 4:35 pm Post subject: |
|
|
| UrgentJensen wrote: | Morning,
In my experience the worst CM related stories are when a major change is planned and looking good, but shortly afterwards it becomes apparent that all is not well.
For example, once we were having a data centre power down and whilst that was going ahead a couple of the network guys thought it would be the ideal time to upgrade the config on the core switch. Trouble is that switch was single point of failure and the entire company completley disappeared off the radar for a while.
Oh, and this is my favorite: Due to pressures a customer on a poorly run project I found that one of the techs had decided he couldn't wait for slot to be allocated in the server room for his new box and we found out a while later that he'd put it live... lying on the floor in a closet. With four other servers stacked on top. F&*king unbelievable.
Oh the trials and tribulations...
UJ |
Reminds me of my last job. The server room was over full, power and air cond resources were stretched beyond capacity, the floor was bending. And edict was issued From On High that there were to be no changes at all made to any equipment in the server room without the approval of Facilities and Operations. Several new projects were coming on line. Solution? Use the room NEXT TO the server room to put the new servers in. Was it secured? Nope. Did it have air cond? Nope. Don't ask me what happened, I'm not there any more
I started in Operations many many years ago in the days when it was normal for stuff to work one night but not the next and after the programmer abused you for calling him up at night would say oh yeah I changed that today, and that is why I am a change manager |
|
| Back to top |
|
 |
UKIT Itiler

Joined: Sep 26, 2007 Posts: 36 Location: England
|
Posted: Mon Dec 21, 2009 4:53 am Post subject: |
|
|
| With reference to datacentre issues,such as lack of space,power and cooling issues.This is an area that appears to be always suffering from lack of management,having worked for some leading IT companies over the past 10 years or so.With server's being smaller than ever but drawing significantly more power than they predecessors the upgrading of the datacentre's infrastructure is achieved through re-active rather than proactive measures in some cases. |
|
| Back to top |
|
 |
|