Problem Resolve tool

[Banan]

Hello!

First post.

I started working as a Problem Manager for a large account this summer. At any given moment we have around 400 open problem records, our team consists of 10 people, and we deal with about 100 diffrent systems.

ITIL and Problem Management is very new to me, I have worked as a software developer for the last 10 years, but wanted to try something new. I still have a _lot_ to learn, but have not regreted my decision a single moment.

Anyway, let me get to the point(s):

Can you recommend any tools for problem determination?
Has anyone come up with a "Cheat-Sheet" for problem management?
And just for the heck of it; what is your definition of an excellent Root Cause?

Thanks, and have a nice day.
Anders.

[JoePearson]

OK ... Problem Management day.

Hi Anders - welcome!

I'll assume that your organisation has a clear and ITIL-consistent distinction between incidents and problems. If that's not the case, post back.

Do your responsibilities include resolving problems - or managing the process?

Problem determination tools and cheat-sheets are usually technology specific, and focused on resolution rather than process. There are some problem determination techniques, like Ishikawa diagrams (start with Wikipedia), that are generic and useful - especially for those most pleasant of problems, the inter-disciplinary ones.

But I hope you're managing the process - logging and tracking at least - because if no one is, then your company isn't doing problem management.

The first thing you should do is some analysis of common characteristics of problems - are particular services, technologies, departments, etc commonly involved. Those areas might benefit from p-d tools and cheat sheets (even standard diagnostic scripts or knowledge base articles that can go back to the Service Desk).

And ... root causes. A root cause is the cause which, if rectified, will prevent the problem from recurring, in the most efficient way and with the widest scope. By most efficient I mean you can't get a greater quality improvement by going back to the cause of the root cause (everything is caused by something else, but eventually you get diminishing returns. And by widest scope I mean you don't just fix the known error in one place, you fix it everywhere.

An example which I picked up years ago:
- a service outage occurred
- the outage was caused by an apparent server crash, but the crash was in fact a controlled automatic shutdown
- the shutdown was caused by a thermal overload being detected
- the thermal overload was caused by the server getting too hot, which was caused by the room getting too hot (possible causes relating to the server fans and the rack fans being faulty were eliminated)
- the cause of the room getting too hot was a failure of a room temperature sensor (possible cause relating to the aircon units was eliminated)
- the failure of the sensor was due to not following prescribed maintenance checks (not a product quality problem)

By ensuring that maintenance procedures for all environmental items are correctly followed (in all server rooms!) a greater and more sustainable prevention of future problems is achieved than if the aircon was serviced, or the single sensor replaced.

[jpgilles]

thanks Joe: that's a wonderful example of what a root cause is....

However, let me challenge you on this: just for fun and also to highlight some issue with cause determination and how far to go....

The reason why prescribed maintenance checks were not followed was because the maintenance folks were not serious at work and spent more time talking than working.

The reason for this was:
option 1) for what they are paid, they think what they do is enough
option 2) HR insisted on selected somebody with broader skills than required so they would be capable of evolutions and the guys got bored.

1) the change required to fix the situation in option 1 would be: upgrade their salary
2) the one required in option 2 would be : fire the HR department head

A common solution would be : get that service outsourced to a third party with signed SLA where they would have to pay penalty fees is the situation happens again....
_________________
JP Gilles

[JoePearson]

But the maintenance was already outsourced. The outsource provider management underpaid its staff and promised them greater job satisfaction so both your 1 and 2 apply. Also, the staff were not as motivated in the success of the client company as the previous in-house staff were. The outsource company invested most of its resources in lawyers to write tight contracts and fight any claims for penalty pay-outs. Root cause: bad contract. Possible solution: in-source those maintenance functions!