Diaxion Blog, IT Governance, Server Virtualization, IT Infrastructure Strategy

What does capacity planning look like within your organisation?

In most organisations existing capacity planning methodologies are rooted in decades of traditional physical server deployments and dedicated infrastructure. If an application server is at 50% load today, and load has historically doubled every 24 months, chances are your capacity planning methodology predicts that you have two years before you must add further capacity. While such an approach may work acceptably when dealing with dedicated, physical server instances the now widespread use of production server virtualisation limits how accurate and therefore worthwhile such predictions can be.

Further, when this approach fails, IT managers and administrators typically fall back on intuitive approaches to capacity planning –  responding to reports of application slowness, or to changes in headcount, in a linear manner that does not account for the complex relationships between application performance and each layer of the infrastructure upon which the applications are hosted.

These intuitive capacity planning methodologies are at best inefficient, resulting in needless or poorly targeted infrastructure investment. At worst, they can be completely ineffective, resulting in highly approaches to infrastructure management with significant operational costs.

Virtually Unknowable

Virtualisation – along with the adoption of other shared systems, such as clustered database and web servers, hardware load balancing appliances and storage area networks – necessitate a holistic approach to capacity planning. It is no longer enough to simply understand resource utilisation on an application-by-application basis. Instead, IT managers must consider the inter-relationships between applications; when are peak periods for individual applications, which applications have peak periods which overlap, how do applications map to line-of-business functions, etc . Each additional piece of data which must be included in capacity planning calculations exponentially increases the complexity of the forecasting, increasing the likelihood for error and therefore decreasing the value of the capacity planning exercise itself.

Given this it is no wonder that in most organisations capacity planning for virtualised environments remains an ad hoc process, with virtual infrastructure administrators applying the traditional physical server cap planning methodology to ESX hosts and simply trying to manage around its shortfallings via “agility” in infrastructure procurement and deployment.

A New Approach

A new generation of tools are beginning to emerge that seek to resolve these problems. Approaches vary across vendors but we can see common themes between them:

• the ability to automate application mapping, allowing analysis to incorporate relationships between servers
• the ability to rationalise performance and capacity metrics from multiple infrastructure layers – typically application, database, operating system, hypervisor, network and storage
• Scenario-based modelling of growth

By automating discovery and data collection and by operating across all layers of the application/infrastructure stack these tools help drive a transition from the old, intuitive capacity planning methodologies to one that is based on hard data, and therefore much better able to accurately predict capacity demands within your unique environment. And, as we will discussing in a forthcoming post, such a data-driven approach is critical to managing not just capacity forecasts but application performance as well.

Does your organisation have a true performance management methodology? For the majority of organisations the answer is simply “no” – performance management amounts to a variety of disparate, ad hoc and predominantly reactive  processes. Examples of such approaches may include:

• Server utilisation monitoring – perfmon statistics – CPU/memory/disk utilisation etc.
• CMDB
• User-feedback – “It seems slow”
• Transaction response time monitoring (“stopwatch testing”)

Common limitations of these legacy approaches include:

• Not data-driven
• Reactive – bottlenecks are typically only identified after they cause performance problems
• Don’t take into account shared systems – virtualisation/SAN/network
• Obtaining more useful data requires significantly greater operational investment
• Tools tend to be focus on individual infrastructure layers making it difficult to build processes that are useful across the entire enterprise infrastructure
• Baseline performance benchmarking only useful for before/after analysis – cannot be used for accurate “what if” scenario planning

The largest challenge faced by infrastructure administrators in responding to performance problems is a lack of data. When users complain that “it’s slow” administrators lack the critical information needed to effectively respond – how did the application perform before the issue arose; what utilisation metrics correlated to the previous, acceptable, performance level; to what degree is performance now degraded; what has changed between then and now?

As a result, administrators tend to fall back on intuitive approaches to performance troubleshooting – looking for errant performance metrics, reviewing code release schedules and recent infrastructure changes, and frequently fall back on crude techniques such as increasing available computational resources in a hope to resolve performance bottlenecks. Such approaches are inefficient and are not cost effective, and do not scale to large, complex environments.

A new wave of tools is emerging that seek to resolve these problems. These tools are generally “cross domain”, referring to their ability to collect and analyse data from multiple infrastructure layers. Typically, the include the ability to determine whether performance variations are due to increased load, code changes, infrastructure changes, or impacted by performance of shared system components (ie, where an application’s performance is degraded due to increased load on a shared component such as a virtualisation farm).

An additional benefit of a data-driven performance management approach is the ability to “right-sizing” infrastructure – particularly in virtualised environments, resources are often over-allocated to individual servers and are therefore wasted. Once administrators fully understand the true performance and resource requirements of applications, these wasted resources can be reclaimed and reallocated.

In addition, they are capable of complex scenario modelling – this allows administrators to forecast the impacts associated with, for example, a successful web marketing campaign, the hiring of 100 new office staff, or the opening of a new branch office. As a result administrators can proactively identify future performance bottlenecks and IT infrastructure spending can be targeted to where they will deliver most benefit. Further, by understanding application utilisation trends and knowing  where bottlenecks reside in their infrastructure, administrators are able to reesolve performance issues before their users even notice them.

Performance and capacity are topics often discussed in a virtual environment, but understanding is often limited to what can and should be measured and how capacity planning takes place. There is an obvious inter-relationship between capacity and performance (and availability). It is quite common to see clients with alleged performance issues but using the basic metrics of CPU and memory usage as the (only) key indicators.

Ultimately, what is important is application delivery, productivity and usability by the customers, as this is essentially the reason for being for computer systems. Applications are typically managed by application teams and systems by systems teams. It follows that service requests (around performance) are primarily initiated by these end users who have visibility of application responsiveness and usability in general. The reality is that capacity shortages cause a large amount of all outages, and often has a direct relationship with performance.

One (non-IT) related example I’ve always remembered was during my engineering days visiting an aluminium smelter. The electricity supply (capacity) was so crucial to the business that a lack of supply for more than a few minutes meant the potlines would solidify, and cause an outage of between months and years.

In the virtual environment, the upside is that visibility of the key metrics is much simpler because of the shared nature of the technology, and visibility is of course a key to management. The VMware client readily exposes fundamental items such as datastore used/free space, custer/host/VM memory and CPU utilisation which we are all familiar with. Under the hood, the performance counters (and hence API’s) expose many more metrics (around 150 in total).

Some key ones are as follows:

CPU ready (cpu.ready.summation) – The amount of time spent waiting for a CPU (core) to become available. With ready times, VMware presents this in milliseconds, whist using esxtop displays as a percentage. This sometimes causes confusion, but the conversion is straight forward: simply divide the value (say 3,500) over the number of milliseconds in the interval (20 seconds @ 1000ms) and multiply by 100 : (3500/(20 * 1000) ) * 100 = 17.5%.

CPU usage (cpu.usage.average) – Expressing CPU utilisation as a percentage of the total presented resources (i.e. for a 2 vCPU machine, 100% would represent full utilisation of both vCPUs, but not necessarily the same 2 physical host cores). This is what is visible in the VI client.

Memory swap-in (mem.swapin.average) – The rate at which VM memory is reclaimed from physical disk

Memory swap-out (mem.swapout.average) – The rate at which VM memory is put to disk. Both swap in and swap out are excellent indicators of insufficient host memory, more so than just swap utilisation.

Memory usage (mem.usage.average) – This is what is displayed in the VI client, and is expressed as a percentage of granted (assigned) memory.

Disk read latency (disk.totalreadlatency.average) – The round trip time (in milliseconds) from ESX to the platter for a read request to be serviced.

Disk write latency (disk.totalwritelatency.average) – The round trip time (in milliseconds) from ESX to the platter for a write request to be serviced.

Both read and write latency is a good indicator of storage health, but should never be used as the sole indicator, and this holds true for all performance.

One important thing to note is when looking at performance is regarding clusters – the VI client and API both present CPU and memory objects for Clusters as well as Hosts. Reporting on cluster performance is simply an aggregate of each host currently in the cluster, so this will skew depending on what host is currently present in which cluster. This will have a drastic impact on historical reporting on cluster performance if the cluster nodes are changed significantly or frequently.

The VMware acquisition of B-Hive in 2008 was no doubt to provide a higher and more orchestrated management approach to application performance, rather than simply systems performance and to align those performance characteristics with SLA’s. The big picture is portraying a virtual world where we have increased visibility and understanding of performance and the relationship to the physical hosting infrastructure to help us plan, manage, integrate and report better.