ROI Methodology®

The ROI Methodology is the most recognized approach to ROI evaluation. This methodology is implemented in over half of the Fortune 500 companies as well as several other private and government organizations throughout the US and in 70 countries internationally. The
ROI Methodology provides organizations a process that can cut across organizational boundaries, linking programs, processes, and initiatives to bottomline measures.

The ROI Methodology has sustained its position as the leading approach to program evaluation because it:

Reports a balanced set of measures
Follows a methodical, step-by-step process
Adheres to standards and a philosophy of maintaining a conservative approach and credible outcomes

A Balanced Set of Measures

The concepts of cost-benefit analysis and ROI have been used to show the value of programs, processes, and initiatives for centuries. Cost-benefit analysis is grounded in welfare economics and public finance; ROI in business accounting and finance. Together the two are the ultimate measures of the contributions of programs, processes, and initiatives. But alone, they are insufficient. While cost-benefit analysis and ROI report the financial success of programs, they omit critical evidence as to why the financial impact is as it is. By balancing financial impact with measures that address individual perspectives and the systems and processes that support the transfer of learning, a complete story of program success can be presented.

The ROI Methodology categorizes evaluation data into five levels. These five levels tell the ultimate story of program success.

Level	Measurement Focus
1. Reaction & Planned Action	Measures participant satisfaction with the program and captures planned action
2. Learning	Measures changes in knowledge and skills
3. Application & Implementation	Measures changes in behavior and specific actions on the job to make the program successful
4. Business Impact	Measures changes in business impact measures
5. Return on Investment (ROI)	Compares the monetary benefits to the costs

Level 1: Reaction and Planned Action

This initial level of evaluation is the most commonly used; reaction and planned action data are usually collected by an end-of-course questionnaire. This is the questionnaire typically offered to participants at the end of all programs. Although criticized by researchers because of the limited relationship between participant satisfaction and the use of the knowledge and skills, when used appropriately, the Level 1 Reaction questionnaire is a source of data that predicts actual use of skills and knowledge. Level 1 evaluation answers five questions:

Is the program relevant to participants’ needs?
Is the program important to participants’ success?
Did the program provide participants with new information?
Do participants intend to use what they learned in the program?
Are participants willing to recommend the program to others?

All five of these measures usually show a significant correlation with application.

Level 2: Learning

Every program has a learning component. Acquiring new knowledge and skills is necessary to change behavior in order for a participant to apply what they have learned. Learning measurement takes place during the program through a variety of techniques such as tests, facilitator assessment, peer assessment, self-assessment, observation, reflective thinking, and documentation. Learning measurement answers two critical questions:

Do participants know what they are supposed to do and how to do it?
Are participants confident to apply their newly acquired knowledge and skills when they leave the program?

Level 3: Application and Implementation

Executives and managers often make this comment, “It’s not what the employees learn, it’s what they do with what they learn that’s important.” Organizations invest millions in training and performance improvement initiatives, yet there is still limited evidence as to what is gained through those initiatives. Measuring application and implementation provides evidence that the learning is transferring to the workplace; that employees are doing something different because of the learning.

Success in application and implementation is measured after participants have applied the knowledge and skills on a routine basis, although it can be forecasted using a variety of techniques. Data are collected through surveys, questionnaires, interviews, focus groups, observations, action plans, and performance contracts. A critical component of application and implementation measurement is the determination of enablers supporting the transfer of learning and the barriers preventing the transfer of learning. Measurement at Level 3 provides the richest source of data. Five key questions are answered when measuring success at Level 3:

To what extent are participants applying their newly acquired knowledge and skills?
How frequently are participants applying their newly acquired knowledge and skills?
How successful are participants with the application?
If they are applying their knowledge and skills, what is supporting their effort?
If they are not applying their knowledge and skills, why not?

Level 4: Business Impact

Measuring business impact connects the program to important business measures. Improvement in productivity, quality, cost, and time is critical in any organization. Other measures of success such as customer satisfaction, employee satisfaction, and innovation are also important in achieving organizational goals. Level 4 Business Impact measures are defined as the consequence of applying skills and knowledge from the program. Measuring these consequences answers two questions:

What is the business connection to the program?
What is the impact of the program in the organization?

Level 5: Return on Investment (ROI)

The ultimate measure of the financial success of a program, process, or initiative is ROI. ROI compares the monetary benefits to the costs of the program. To calculate ROI, six steps of cost-benefit analysis are taken:

Step 1 Identify the improvements in impact measures.

Step 2 Isolate the amount of improvement to the program.

Step 3 Convert the improvements to monetary value.

Step 4 Tabulate the fully loaded costs of the program.

Step 5 Identify the intangible benefits linked to the program.

Step 6 Compare the monetary benefits to the cost in an ROI calculation.

The first step, improvements in business measures are taken when evaluating at Level 4. The amount of improvement connected to the program is identified (step 2). These improvements are then converted to monetary value (step 3). A fully loaded cost profile is developed (step 4) and intangible benefits are defined (step 5). Finally, the monetary benefits of the program (Level 4 measures converted to monetary value) are compared to the fully loaded costs of the program to calculate an ROI (step 6). Measurement at Level 5 answers one critical question:

Do the monetary benefits of the program meet or exceed the costs of the program?

When fully developed, these five levels of evaluation plus the intangible benefits represent a chain of impact that occurs when participants attend a program, learn, apply their learning, and improve impact measures. Figure 1 depicts this chain of impact. As shown, the chain of impact represents the five levels of evaluation along with a step to isolate the effects of the program. This is an imperative step in the process that answers the question:

How do you know it was the program?

The Chain of Impact Tells the Complete Story of Program Success.

Not all programs are evaluated at all the levels. Only certain programs require such comprehensive evaluation. Those programs include very expensive programs, programs that have a long life cycle, programs that have a broad reach, comprehensive programs, and programs that instill significant change in the organization. Other variables such as the need for the program, the purpose of the evaluation, and the stakeholders’ needs often drive the level to which a program is evaluated.

ROI Methodology® Process

The ROI process integrates design thinking principles into four process stages to ensure the appropriate data are collected from the proper sources at the right time. The process consists of sequential, logical steps that lead to data categorized by the five levels of outcome data, using design thinking principles to deliver business results. The process of evaluating a program begins with asking why, then aligning programs to business needs, and concluding with optimizing the results. This section provides an overview of the ROI process.

Data Collection Procedures

The data collection process includes four primary elements: 1) defining the program objectives and measures to determine if the objectives have been achieved; 2) determining the data collection methods; 3) identifying the sources of data; and 4) determining the timing of data collection.

Program Objectives and Measures

Program objectives are typically derived from a formal needs assessment. The needs assessment identifies the gaps in performance. The gaps can come from excessive costs and inefficiencies, behavior, understanding, and even individual perceptions. Once these gaps or needs are identified, a solution is developed to solve the problem. To ensure that the solution designer, facilitator, and participant involved in the process understand the expected outcomes, clear objectives are defined. The solution is then evaluated against those objectives. The needs assessment drives program objectives, which then drive the evaluation.

Data Collection Methods

A variety of data collection methods are considered when planning an evaluation. These methods include tests, surveys, questionnaires, interviews, observation, focus groups, action plans, and organization records.

Data Sources

Selecting the data source is a critical step in data collection because the source drives the credibility and validity of the study. The fundamental question asked when selecting data sources is:

Who knows best about the measures being taken?

Timing of Data Collection

Data are usually collected in two different time frames: 1) Level 1 and 2 data are collected during the program, and 2) Level 3 and 4 data are collected after participants have had time to apply knowledge and skills on a routine basis.

Data Analysis Procedures

Data analysis begins with isolating the effects of each initiative from other influences that may positively or negatively impact Level 4 measures. When ROI is calculated, the benefits of the program (Level 4 Impact measures) are converted to monetary value and compared to the fully loaded costs of the program. This section presents options for data analysis.

Isolating the Effects of the Program

A variety of techniques are available for isolating the effects of a program:

Control group arrangement
Trend line analysis
Forecasting methods
Estimations
Previous studies
Subordinate report of other factors
Calculating/estimating the impact of other factors
Use of customer input

Appropriate techniques to use in a survey are determined during the evaluation planning phase.

Converting Data to Monetary Value

Data conversion takes place only when the ROI is pursued. This step is dependent on the Level 4 Business Impact measures that improve due to the program. Not all Level 4 measures need to be converted to money; some will be reported as Intangible Benefits. The techniques that can be used to convert measures to monetary value include:

Using standard values for
- Output
- Quality
- Employee’s time (calculate the value using)
Relying on historical costs and records
Locating internal and external experts
Data from external databases
Securing estimates from
- Participants
- Supervisors/managers
- HR Staff
Linking with other measures

This step in the process will be implemented as important business impact measures are identified through evaluation planning and data analysis.

Tabulating Fully Loaded Costs

The final data necessary to calculate ROI is to determine the fully loaded costs. These costs include needs assessment, design/delivery costs, facilitation, participation, and evaluation costs, representing a fully loaded cost profile necessary to ensure a conservative estimate of program costs.

ROI Calculation

The ROI calculation is a standard financial equation developed through finance and economics:

BCR = Program Benefits
Program Costs

ROI (%) = Net Program Benefits
Program Costs

Intangible Benefits

Along with the ROI calculation, intangible benefits of each initiative will be identified. In many cases, these intangible benefits, the benefits not converted to monetary value, are more important to an organization than a positive ROI. Intangible benefits may include enhanced job satisfaction and commitment, customer satisfaction, teamwork, images, stress, and job engagement.

Throughout the evaluation of each initiative, value will be assessed at every level. Data will be provided to the client to ensure that improvements to the initiatives are implemented. A comprehensive report will be developed that fully describes the value of each initiative.

Scope and Use of the ROI Methodology®

The scope and use of the ROI Methodology have grown tremendously. The methodology had its beginnings in the learning and development area and quickly grew into other applications. The process has also expanded to all types of projects and programs in a wide variety of organizations including non-profits, NGOs, and the public sector. The process has been fully implemented in more than 70 countries and the list is growing.

This process captures a balanced set of data. It is systematic, user friendly, and brings business evaluation to major programs. The process is supported by a tremendous amount of research with practical usage and application. It is estimated that 3,000-5,000 studies are conducted each year. The conservative standards make it executive friendly. The ROI Methodology has become the most documented evaluation system in the world with books in 38 languages and case books developed in at least a dozen countries.

The Benefits of ROI

The payoffs of the ROI Methodology are numerous. This approach satisfies the desire to have a credible process and shows the contribution of important programs. Perhaps the four most important reasons are: to improve programs (process improvement versus performance evaluation for the program team is critical), gain support for the programs, build relationships with key executives and administrators (who often fund the programs) and provide funding for programs in the future.

ROI Methodology: The Payoff