There's a familiar French proverb that says, "Nothing succeeds like success."
But it's difficult to get the benefit if success is not defined, measured and built on. This is particularly true of information technology (IT) projects. Most organizations find it troublesome to identify the criteria that represent a successful IT project outcome. Consequently, they are inconsistent in managing to these criteria or evaluating the results.
That's why researchers Graeme Thomas and Walter Fernandez of Australian National University recently undertook a study to examine how organizations in Australia define IT project success. They also wanted to know which methods are most effective, and why.
What they discovered: IT project outcomes do improve when organizations define and measure their success criteria.
The researchers studied 36 Australian companies operating in three industry sectors: finance and insurance; electricity, gas and water supply; and mining.
They found that just one-third of these companies had formally defined success measures. These companies, they found, also had higher levels of overall confidence that their IT projects were producing benefits.
Defining success
Companies in the study categorized success into three broad areas: project management success, technical success and business success. While the study found 14 success criteria in use overall, the average company used only five. These included the standard notions of ‘on time' and ‘on budget' but also included ‘delivery of benefits', ‘met business objectives' and ‘business continuity' (the level of disruption that an IT project has on operations). Other criteria used included ‘sponsor satisfaction', ‘project team satisfaction' and ‘steering group satisfaction.'
One-third of companies used a formal success definition while another third had no formally agreed success construct.
"There was recognition among more thoughtful companies that it's possible to have project management success without business success, and vice versa," notes Fernandez. "Success was more than just meeting the requirements in the business case."
Overall, companies with a high level of confidence in their success also had a formal success definition that was widely understood and accepted; used a balance of about five success criteria, drew a clear distinction between project management success and business success, and had a clear focus on delivering benefits.
Measuring success
While 75 percent of all companies studied did attempt to measure success, only five companies were highly effective at doing so.
In fact, most companies indicated that identifying and measuring the costs and benefits of their IT projects is a major challenge. What's more, just over half of companies said they identify all benefits while 28 per cent said that they adequately measure them.
Only the five companies that clearly defined their success criteria went on to measure for success using a post-implementation review, a benefits realization process or a combination of both. They then tracked benefits for six to 12 months after implementation. The researchers found that using these formal systems, rather than just relying on perceptions, also helped to manage the tension between performance incentives and the desire to over-report success.
"Consistent measurement of success provided the basis for improvement of both project and benefits delivery," notes Thomas.
In other companies, post-implementation reviews only occurred ‘when things went wrong'. Reasons for not conducting post-implementation reviews or benefit realization included lack of management support, unclear ownership of processes, limited accountability, resource constraints, difficulties with measurement and attribution and inadequate use of evaluation results.
Applying success
The researchers discovered that those companies with a high level of confidence in their IT projects not only agreed on a definition of success and then constantly measured it; they also used the results. This included managing the project according to the agreed definition, a willingness to stop projects, accountability for results and a connection to learning.
The most effective evaluation practices were found in the companies that held their business managers accountable for results. Accountability was enforced through formal project reporting, performance incentives, individual appraisals and department budgets.
Nor was measuring success solely to the benefit of the one project. These companies used the results to continuously improve project management and estimation and also reinforced the use of post-evaluation practices.
"Our research clearly demonstrates how important it is to capture what success means to an organization in order to achieve success," says Fernandez.
Recommendations for success
- Obtain wide agreement on the definition of success as early as possible. Recognize that it may have multiple elements.
- Measure consistently against your agreed definition
- Apply results to better manage progress and performance, using rigorous accountabilities.
"Quite simply, says Thomas, if you know what you are looking for and if you track your progress and are willing to alter your path, then your chances of success will be better".
Contributing authors: Dr Walter Fernandez and Mr. Graeme Thomas
PMPerspectives.org is a website which connects project managers and sponsors with project management researchers. Our mission is to understand and improve project management practices. The research team comprises Dr. Blaize Horner Reich and Dr. Andrew Gemino from Simon Fraser University, Canada and Dr. Chris Sauer from Oxford University, UK.
© Reich, Gemino, Sauer (2008)
