Reduce Incidents Caused By Change

[JDYE]

Hello,

I'm just a newbie so be kind to me.
We are a large IT business and have been using ITIL standards for Change and Problem mgmt. for sometime now. We require Controlled work instructions with Peer Reviews but we still have a high number of incidents caused by change.
Simple story is that most are caused by human error. People not following the work plan or sleep deprevation might be the cause. Could be that people are just not following their work plan 100% and/or "fat fingering" their typing.
I have researched the web and can find tons of info on HOW you "should" manage change but I can find nothing on how to reduce the number of incidents caused by change. Expecially when human factors are involved.

I guess I'm just looking for common sense advice.


Thanks !

[UKVIKING]

JDYE

First questions:

How do things go to Production ?

a) they are tested first using the same instruction set in a non production environment by the same team / individual that will do it in Production

b) it is done for the first time in Production

Are these done as Emergency Fixes and Rushed to be done ?
_________________
John Hardesty
ITSM Manager's Certificate (Red Badge)

Change Management is POWER & CONTROL. /....evil laughter

[Timo]

Wow... you almost sound kind, John.

I had same question - how do changes get into prod? To add to John's, do same people deploy changes to prod and produce and test (presumably) rollout work instructions? If yes, there will be propensity to make shortcuts.

Have you tried the threat of physical violence technique?

[JDYE]

Well, without telling too much of myself on the Internet I am 6'4" and 260lbs so yes I have thought about "physical" compliance. (jus kidding) !

We do have some test and pre-prod environments where some changes can be tested. Our systems are very complex which makes it hard to have a test environment for everything especially Network and SAN.

The change requestor is the same person performing the change (99% of the time) and doing the testing were possible.

All planned changes with a moderate or major impact must go through our CAB. All Standard and Normal (minor) changes are self- approved with a work instruction and peer review. So at the very least even the simplest change would have 2 people review the work instruction prior to implementing.

I hate to tighten the screws on technical peer reviews but if mistakes are taking place then people are not following the WI or there are errors in the WI.

[Timo]

I hear you about the complexity of the env't and testing... it's a pretty common problem I think.

Well... I don't know if there is one answer that will solve your problem but with one client we did significantly reduced incidents caused by moving changes into production by segregating build and test roles from deployment roles. Most of deployments happened at night, making people responsible for deploying less than happy... hence the deployment team went an extra mile (more like harassment toward developers and testers) to ensure that the instructions and packages they were given would work, so that they wouldn't have to stay up any longer than they had to.

Personally, I love nothing more than tightening screws on techies and developers but they don't pay attention to me cause I am "the process guy". When their other technically inclined peers apply the same pressure it has different effect. It's kind of a soft stuff, but in our example it worked.

I am sure others (when they wake up) will have different suggestions.

[Diarmid]

The only way to reduce the change related incidents is to investigate the occurrences. If there are quite a few then there should be some underlying pattern that will point to what needs to be remedied.

Perhaps people are under pressure, perhaps instructions lack detail (or have too much detail), perhaps their is a training need, perhaps the environment is overly complex, perhaps there are inadequate controls on the configuration or operational processes.

One thing for certain, attitude (or culture) is important. Do the staff understand (in their gut or heart ) the importance of the transition activities? Do they see it as a chore before getting into their next exciting project? Are development and operations different countries with border disputes or is everyone sailing the same ship in the same direction?

Try going over the details with the people making the mistakes and ask them how things could be different to prevent such errors in future. Make sure everyone understands that the quality of the release process is as vital as the quality of the development.
_________________
"Method goes far to prevent trouble in business: for it makes the task easy, hinders confusion, saves abundance of time, and instructs those that have business depending, both what to do and what to hope."
William Penn 1644-1718