Capacity Mgmt Metrics

[SwissTony]

Been having a confrontation with a newly appointed associate director who has recently been involved in conducting an ISO20k internal audit.

Within the report he has stated that CPM is failing as no 'processing capacity' metrics are monitored, measured, and reported preventing correct decisions to be made.

Now my issue here, which I would appreciate you thoughts & comments on are, we monitor CPU & Memory usage with Ganglia (previously using Orca). We have real time access to the information, however, choose not to measure the CPU or Memory levels of 800+ servers (mixture of windows, linux, aix, unix, physical, virtual etc). My justification is that we recieve alerts when thresholds are reached so that the users are not affected - that when new applications / services are being added the guys have access to this information to judge whether or not to add it to box X. Finally, that to measure CPU & Memory over so many servers to make it efficient the data would have to be culmiated into so sort of average, overall figure, which then makes it completely in-effective.

Do you guys measure 'processing capacity'? & How do you do it?

[Diarmid]

Tony,

if you do not know the present average and peak utilization of a machine, how do you determine its capacity to support a new application, a modified application with new functionality, a change in utilization patterns. The absence of threshold alerts is no guarantor of available capacity for additional usage.

Performance monitoring is not Capacity Management although it is a necessary component.

Now the practical aspect is, as always, cost and risk. What is the cost of maintaining current baselines for each machine? What are the consequences if a particular machine "overloads" (on the busiest day of the year, for example).

So it can be okay to make a judgement call. Especially for small stable systems running non-critical systems. It can amount to how tolerant your business activity is of small glitches in responsiveness occurring for a period;how readily you can either add capacity or reshuffle apps between boxes; how acceptable it is to regress and move a system newly added when the capacity is found to be wanting.

But consider this scenario:

A project is proposed to revamp a cluster of applications that are spread over 20 small servers. This will be a big deal. Therefore, either:
- you set up life size tests to measure everything as it will be
- you measure the new apps and apply modelling techniques (or at least extrapolations although that is less reliable) to predict impact on capacity
- you make a judgement call (I don't think so)

If the first is too expensive (on people, machines and time) and possibly rather difficult, then you have to look at the second. This is fine. But if you do not have current baselines for these machines you will have to extend the project to achieve these before you proceed (typically that involves working through peak demand periods which may be monthly for example)

So one of the risks is that it might become important in the future. But it is still cost and risk.

Your auditor is only correct if you cannot show the cost and risk analysis to support what you do.
_________________
"Method goes far to prevent trouble in business: for it makes the task easy, hinders confusion, saves abundance of time, and instructs those that have business depending, both what to do and what to hope."
William Penn 1644-1718

[SwissTony]

Diarmid, thanks for your response..

[Diarmid]

[Cking]

Tony,
I've been bouncing around a few sites concerned with measurement and capacity...
One term that often shows up alongside Processing Capacity is Cost of Transaction. (Forgive me if it's in ITIL as well, I've been introduced to alot of terms from many sources lately)

So not only do you need to capture the box metrics, but also usage / throughput / transactional metrics to match up with performance metrics.
... we haven't gotten there yet either.

Then when you know of a change in demand coming down the line (seasonal, new customer, etc) , you can estimate what the increase in transactions will demand from your infrastructure.

For new apps or enhancements, ideally that relationship would be captured during the QA phase.

And Yes, when you roll historical data into averages you lose visibility of reality. So maybe keeping peaks, or average of peaks alongside a variance metric like standard deviation or confidence interval (6Sigma background ) might give a better picture.