Projects and Emergency changes giving me a headache!

[mstickle]

Hi All,

I've read through a few of the posts relating to Emergency changes, but none of them seem to offer a solution to the problems I have been experiencing.

I am the new Change Manager in my organisation and took over from a previous manager who had already implemented the process. I haven't made many changes to the documented procedures, but I am having a hard time with projects and Emergency changes. Here's a scenario:

A new project is delivered after being tested and approved etc...however once it goes into the live environment we seem to get a whole host of issues that were not spotted during testing. Pandemonium breaks out and Emergency changes appear left right and centre; I do my best to keep everyone calm and rational but when I suggest we call a meeting to discuss the ECs I get told that the decision has already been made by the business (basically 'butt out!') I often discover that decisions are made without me being present and I am only told at the very last minute that 'we have an Emergency change to put in and this has been approved by such and such'...however the person often approving the change has no idea about the processes and does not consider important factors such as scheduling the change, informing the relevant people within the business or the customers. They also categorise things as Emergency when in fact they are not!

I see this happening time and again and I am getting rather fed up with the number of Emergency requests that come out of a poorly tested or implemented project - to make matters worse, despite my best efforts to keep the situation under control, other managers seem to override what I've said...

Has anyone else come across this situation? I've requested a post implementation review and issued a report on the number of emergency changes that we've had as a result of this to senior management, but it seems that each time people become apathetic to the situation and so it starts all over again. I'm not an experienced manager let alone change manager so perhaps I'm not approaching this in the right way - an tips?

[ARoll]

First and foremost, what level of upper management support do you have for the change process? An approach would be to build some data around these releases and correlate the number of EC's that are resulting. Present this information to the power's to be along with your suggestions on how to combat the problem and get their buy in. Show the results to them in terms that will peak their interest ( $$ lost due to rework, manhours spent fixing etc).
_________________
Adam
Practitioner - Release and Control
Blue Badge

"Not every change is an improvement, but every improvement requires a change"

[dboylan]

Your issue lies with a lack of authority. As the Change Manager, you should have the ability to veto any suggested change. If changes are being implemented without your approval, then those implementing the change should face disciplinary actions. Once the people who are implementing the changes know that there are repercussions to implementing changes that you haven't signed off on, the practice of "The business has approved it so do it" will quickly stop.

[joeblough]

You have not mentioned whether or not you have a CAB or ECAB setup as part of your Change Management process. It seems like you are missing that important aspect of change management.

You need to kill one to scare the rest. So make an example out of someone who is not adhering to the process. You really need to have the blessings of upper management to put this in place.

A very important factor. Do you have Release management process setup? If so, make sure they do not release anything without approval from your group. Problem solved

[Ed]

I absolutely agree with Adam, Don and Joe.

Ensure that Senior Management is aware of the cost to the business. Get the buy in of the IT Director. Your process must involve a policy statement that makes it quite clear that any introduction of an unauthorised change will be seen as a disciplinary offence. If this is not in place, get it there pronto. Once it is there, publicise it extremely loudly, then bite the bullet and as Joe puts it kill one to scare the rest i.e. take it all the way to a verbal/written warning. I know that this is harsh, but sometimes it is the only way to get the cowboys to sit up and take notice.

Regards

Ed

[pac666]

Make sure you include the % of Emergency Changes in your CM metrics reports. That way you can track which way it's heading and also ensure you shout loud that one of the goals is the reduction in the % of changes that are emergencies.

A high level of Emergency Change is a sound indicator that the organisation is in the throws of chaos. Sell this to Senior Management who should be wanting to run a steady ship and will start getting nervous about the high levels of emergencies getting called.

Also where I work everyone thinks everything is an Emergency (at least to them), you just need to educate them as to what a 'real' Emergency Change entails.

Sometime you will feel like King Canute holding back the tide, just don't give up

[mstickle]

Thanks Guys - I do issue reports on a weekly and a monthly basis showing the number of Emergency changes. I also make a point of raising the number of emergency changes that we have in our CAB meetings.

I did raise this with management today and to be honest they have shown concern as the figures for the past month have been much higher than previous months. However what we don't have at the moment is a cost per change model which I guess is pretty important. I'm sure that once I put this together and management see it in terms of money, then I'll generate more interest. The problem really stems from a lack of understanding within management - I've had to point out the obvious, that this many Emergency changes really isn't a good sign.

As for Emergency CABs well our procedure does mention this, however the terms surrounding it are all a bit vague. This may be part of the problem so I've decided to rewrite the documentation altogether.

The other main issue is the fact that the ECs have come from poor development and planning. Sadly even though we have people overseeing the release of these changes, there is no formal release process in place. I've seen some terrible release notes, often with incorrect version numbers, no back out instructions etc... I've also found changes due to be implemented with a 'we'll suck it and see' approach, because no one is there to investigate the problems thoroughly and realise the changes that actually need to be made.

I liked the idea of making it policy, that if the process is not adhered to then severe consequences will follow. Having put this forward to my superiors... well I'm not sure if this will be considered. I often feel that my position is undermined by the ridiculous amount of managers we have within the organisation and so all too often I'm too low down in the chain to enforce this.

[dam]

I’m more in incident and problem management, but the situation you describe make me thinking to a working experience I had. We were always in emergency mode and the main activity was fire fighting. The main reason was poor application development processes, new versions of the product used to go almost straight from development to live environment, with very light QA and especially practically absent regression tests. This produced a huge overload for services (stormy relations with customers, lots of calls, urgency meetings, lot of time spent on finding and applying workarounds, lots of time on regression fixing releases etc…) . I’ve tried several times to rise the issue with upper management (“Hey boss we waste lot of time in fire fighting, we should improve our app development process ”), receiving always a vague feedback (“I see, yes.. we should of course… we’ll see”).

After a while I’ve started thinking they were joking with me, till when I understood that sometime upper management really don’t have a vision of what happens on the day to day activities, so they really frankly think “people love to complain, they just have to work harder an things will be fine”. Not because they refuse to look at reality, but just because they are not able to look at it.

That’s why you have to collect numbers. And relate your metrics to money, the only things that speak to management. In the situation I described above I started to track any single action related to these regressions, and when after a couple of months I’ve been able to show that we used to spend between 60 and 70 % (!!!) of our time on non billable activities, they started to look at our complains differently. (the upper management: “Fuck! We cannot tolerate this, put your suggestions in a document, I set up a meeting with application dev manager ”).

To resume, you can deal with your staff with a bit of authority (setting up strict rules on changes that could be implemented and applying your veto time by time) but with upper management the only thing that works is numbers in relation with the business.

Having said that sometime even after management became conscious it is still hard to change the existing delivery processes. Some companies are able to work only in emergency mode. There should be a reason why the story I told you is related to a previous working experience…

Good luck!

[OhioScott]

Based upon what I've read, I agree that there sounds like a problem in testing and the organization is not taking adequate steps to address this issue.

This is where you need to stop relying on Change Management and switch over to Release Management. Before any change goes out the door, there should be a Qualitative Plan that lists what the expectations are for the change and what the responses are from testing for the entire system. Depending upon your applications, you could develop an end-to-end test plan where you only need to modify the test plan for the new change and its impacts, thereby having the expected results documented. Anything abnormal should be identified at this time and the customer can decide whether to accept it with workarounds or reject it.

If this type of detail isn't possible... at least do an analysis of past failures to find why they failed and develop a policy or process around this. Don't worry about holding people accountable until you've addressed the blind spot in the process.

Testing is one of those areas that I don't believe ITIL covers well and leaves a lot to the Change Manager and the organization to define acceptable. I know... we have to allow for variation from organization to organization, but something more definitive than 'you should have testing' should be there.