Metrics in IT outsourcing - how it works?

It is a reality that companies achieve cost savings from conventional outsourcing, most of them report a 20 to 50 percent average cost savings over the course of a long-term contract (the typical outsourcing engagement lasts for a number of years). However, the results they achieve depend heavily on the efficiency of the operation to start with. This includes making sure objectives are clear at the outset, paring the performance measures down to a small number of critical ones, shifting from input to output metrics where possible, and making sure these are developed early in the relationship. In every outsourcing arrangement, the customer is concerned with monitoring the functionality being delivered, the development time, and the development cost. The ability to monitor these components is influenced by a lot of factors, including the complexity of the technology, the availability of skilled resources, and the techniques and methods used. The key is to establish an accurate cost per unit of work measure. Metrics are meant to ensure that the client is receiving its required level of service and the service provider is achieving an acceptable level of profitability. Actually, a metric is a progress or quality indicator of the software development.

In theory…

The selection process of the metrics is complicated because they must be tempered by considerations such as organizational experience with metrics, the type of behaviors to be motivated and cost and effort of collection. Often, a metric that works well on one project may be ineffective, inaccurate or too costly to collect on another project. If the wrong metrics are selected, the relationship between provider and customer can go astray quickly.
Also it is good to take in consideration the objectiveness of the metrics. Metrics that are based upon a subjective evaluation are open to different interpretations, and will likely lead to disagreement over whether a service provider has met its commitments.
If the metrics in the SLA cannot be easily gathered, then they will quickly lose favor, and eventually be ignored completely. No one is going to spend an excessive amount of time to collect metrics manually. Ideally, all metrics will be captured automatically, in the background, with minimal overhead.
The first step is the breakdown of the project requirements into subcategory requirements and then into work units for each subcategory requirement. Each work unit belongs to a certain type (simple, medium complex, for example) and is defined by assigning a certain number of characteristics. By classifying these work units, then it is possible to assign a workload which translates into the number of man days for development (or number of hours, depending on the structure chosen).Taking for example a data entry form, the characteristics could be the number of input fields, the number of drop-down list, of images, the complexity of the actions related to the different buttons, etc…This first stage gives a point of reference to the project.
After defining the reference, which is generally used in the first step to accelerate and standardize the estimates, the costs attached to each work unit must be confirmed by correlating it with the actual time spent in developing each requirement. When the reference has reached a certain maturity (equivalence established between the workload and the time spent during development), it is possible to measure the productivity and thus to establish goals for improving the performance of the development team. The idea here is to gradually reduce the workload for each work unit in regards to the reference in order to increase the development speed.
It is better to choose fewer metrics with higher stakes. Outsourcing service providers have significantly narrowed the number of metrics they track over time, increased the accompanying rewards and penalties - in order to minimize administrative demands, and improve their relationships. For example a metric like “engineering efficiency” should be removed from the list of target metrics because it is impossible to quantify the result. It is good to shift from input to output metrics where possible: instead of counting how many hours it took somebody to complete each order, it’s better to count how many orders he completed each hour.
Another good advice is to define metrics early in the relationship. If not, you will waste more time to build the set of metrics to evaluate performance on the way than from beginning.

In practice…

Without going into details, a client asked Pentalog to assure them that we had taken into account a margin of risk in our assessments. The client had indeed noted that we had often exceeded our estimates concerning a precise aspect of the project. Pentalog approached the subject by explaining that it would be appropriate, based on the statistics between the estimated workload and actual workload, to establish a breakdown of the project into work units and assign to each one of them a development load based on these statistics. The concept was discussed. Naturally we discussed the gradual establishment of a metrics system for the project and reminded the client of the benefits that this would incur those advantages that did not fail to catch his attention. Pentalog therefore explained, that by properly defining the work units, the metric system would allow for more reliable estimates and a further increase in the transparency of the method we would use to assign workloads to the functionalities of his product. This added to the fact that he would know in advance the number of days for the realization of a new feature, our client did not hesitate a moment before approving the concept. Furthermore, we also stressed that after the establishment of the metric system, time spent on a subject would be exclusively devoted to the contents of a batch rather than justifying the expenses.
Another example, is the development of a management solution for sales outlets for one of our French customers. The development is carried out by 5 teams spread over 3 Pentalog Offshore sites (Romania, Moldova and Vietnam) with 40 members.
For this project with more than 6000 man days in 2009 based on the SCRUM method (AGILE method), the use of the metrics system is to improve the development productivity by increasing the number of detailed requirements contained in each Sprint; the reference points have been provided at the beginning of the project by the client.
The project manager has to analyze at the end of each Sprint, the work units in order to adjust the development workloads (by increasing them at the beginning in respect to the content of the Sprints and then decreasing them when the team has reached its cruising speed to improve it’s productivity). Having these common references also allows him to measure more accurately the performance of each team and adjust the continuity of the management. The idea is to smooth the performance of all the teams using as a point of reference the faster team and then in regards to the reference development times. After bringing all the teams to the same development level, the project leader can tackle the job to improve the productivity by gradually reducing the workloads of each type.
Indeed, this is a significant advantage for the introduction of the metric system: focusing on the contents in order to optimize them. Of course, the maturity of this system will naturally lead to the optimization of the estimates by better controlling the performance of team members. And from this point of view, everyone wins.