Major Problem / Major Incident

[AndyBostian]

Disclaimer: I apologize in advance if this has already been asked/addressed, but I searched the forum and did not find a similar thread.

The questions are related more to business implementation than they are to ITIL process. I'm curious how others handle these situations in their organizations. Specifically, let's say you have a high-impact high-urgency incident (service down for an extended period of time), root cause is unknown, no solution in sight.

0.) Do you have strict business criteria to declare this a "Major Incident" (and/or "Major Problem"), or is it more of a judgment call by senior management?
1.) How do you successfully continue to run your processes in parallel? (i.e. PM subset of folks investigates for root cause while IM subset of folks looks for workaround).
2.) If you run in parallel, do you have one person with accountability over the whole "war room" situation?
2.a.) Is that person the Incident Manager, the Problem Manager, or another party?
3.) How do you manage resources when both Incident and Problem resolution efforts need the same SMEs? (developers for example).

Thanks for any feedback.
Andy

[Diarmid]

Andy,

Little Rock is a long way from here. So I won't bother throwing stones.

0.) You do both; you have criteria based on, for example, loss (for indeterminate time) or imminent loss of vital services; but senior business management can wade in if in their judgement an incident is threatening the business; part of the issue is often going to be uncertainty of recovery time.

1.) The priority is effective restoration of service; that may well require deep investigation (in effect root cause analysis) but is nothing to do with Problem Management; if there is possibility of recurrence (one way to determine that there is in fact a problem), then the activities (not the management functions) merge into the problem analysis and investigation once resources are free from service restoration; there may not be an underlying problem once the incident is dealt with and the major incident review can check and confirm if that is the case.

2.) The war room is around the incident and, if there is an underlying problem that threatens repeat incidents (for example), especially if they could occur at any time or sooner, then you move the war room to resolving the problem as soon as the incident is fixed.

2a.) In a major incident situation, you do not assign lead role on basis of normal roles; you want the most capable person available running the show; it would be natural for the most senior person present to take charge and s/he may well delegate coalface management to the best and most experienced person available while retaining close contact with what is going on and with senior business management; in any event whoever leads must have authority (at least for the duration of the incident) to acquire and deploy whatever resources are required.

3.) I think this is implicitly answered above; the priority is effective restoration of service.

Problem Management is not a front line activity. If you expect the service to drop again in an hour, or even a day, you are still deep in incident management because you are working on an imminent breach. If you expect the service to stay up but you are not sure whether it might go down again sometime, then you can apply problem management to it and your first step might probably be to validate the workaround that was achieved and streamline its application.
_________________
"Method goes far to prevent trouble in business: for it makes the task easy, hinders confusion, saves abundance of time, and instructs those that have business depending, both what to do and what to hope."
William Penn 1644-1718

[milligna]

This is an interesting topic. I was discussing this with our problem manager here... My understanding of the problem process was that a Problem becomes a Known Error and THEN has a workaround applied. This doesn't fit Incident Management - Incident Management seems to be able to apply workarounds without understanding the root cause in order to restore service.

Does anyone else find it confusing that Problem Management highlights the linear notion of Problem->Known Error->Workaround/solution, and that Incident Management allows Workarounds without understanding the root cause?

Should we have different names for the blind-faith bandaid type of workaround that incident management comes up with (bounce the box!) and the reasoned workarounds that Problem Management comes up with after understanding Root Cause?

[Diarmid]

[UKVIKING]

To add to Diarmid's

You, the user, want your service back. You dont care the reason the service is out. You want it back.

So you try to restore service as quick as possible (Incident mgmt)

Think of it like this

You have a bar full of drinkers watching the SuperBowl or a Championship or Ashes game

You serve (insert beer) on tap. They are drinking it fast.

Your keg empties and you have 20+ beer drinkers wanting beer

Do you a) trying to find the root cause or b) replace the empty keg to restore service

I would do b)

Then the TV reception goes out. (IM part 2). You and your staff frantically try to figure out what's wrong.

You call 'the cable company'
You call your electrician
you call another pub to find out if the incident is only yours or all pubs

One of the bar staff notices a plug is loose and pushes it in and lo you have the game again

still incident mgmt - restoring service

problem mgmt - eitehr reactive or proactive
_________________
John Hardesty
ITSM Manager's Certificate (Red Badge)

Change Management is POWER & CONTROL. /....evil laughter

[Caperz]

AndyBostian - I agree with Diarmid and UKVIKING. I would just like to add that the way we work with this situation is that we have a criteria for creating a problem investigation being a P2 incident record with a high urgency to resolve. This is usually when a large number of users are impacted and the service outage / interuption is impacting business. An example of that would be a line outage to a site.

The key focus is to restore the service to the business, as soon as possible. The speed of the service restoration, in this case, will be dependant on whether the line provider needs to be involved etc. If a simple restart of the modem or NTU restored service then great... otherwise it may be through a reduced performance solution, such as a wireless service (until the line is back up again).

On the problem management side, the reason for outage would be assesed (if problem investigation is warranted). In some organisations and situations it is decided that since the root cause was outside of the internal IT group's control then it is not worth spending the time and effort to pursue root cause with the vendor. This all depends on how your organisation works. In other cases... service disruptions are assesed against the vendor's agreed uptime and action is taken from there.
_________________
ITIL V3 Capability - Operational Support & Analysis Certified