How to Measure Impact and ROI Without Losing Your Purpose

Patti P. Phillips, Ph.D., and Jack J. Phillips, Ph.D

This article was originally published on September 21, 2021, on ChiefLearningOfficer.com.

When thinking about ROI, most people imagine complex calculations, finance and accounting challenges, and misguided measures. In reality, the ROI calculation is a relatively fundamental calculation, comparing benefits to costs in financial terms. The ROI calculation is not needed for most programs. In most situations, other measures reflect the major purpose of why you are implementing a program. However, in some situations, ROI is needed and sometimes required.

ROI is part of a chain of value, as shown in Figure 1. This simple value chain is essential to understanding how value is delivered from any learning and talent development program. While each level of outcome provides a perspective of value, ROI answers the question: Is the program worth it? Did it provide enough monetary benefits (which are usually cost savings or cost avoidance) to cover the cost of the program? All of us make ROI decisions almost daily as we purchase items or services. As you think about buying something, you ask, is it worth this much money? The question reflects a common concern. It shouldn’t come as a surprise when executives, who fund learning and talent development, want to know the ROI of a significant program requiring a large budget and many resources.

Figure 1. Outcome Levels and Types of Data

Level Type of Data
Level 0 Input The number of individuals in the program, how long they are in the program, and the cost of the program.
Level 1 Reaction Participant reaction to the program and its content.
Level 2 Learning Knowledge and skills acquired to make the program successful.
Level 3 Application Applying and using the content in the workplace setting.
Level 4 Impact Either a tangible or intangible measure, often reflected in key performance measures already collected.
Level 5 ROI Return on investment, calculated as a benefit-cost ratio or ROI expressed as a percent.

 

With that backdrop, the remainder of this article will address the four questions we often hear about the use of ROI.

  • Does everything come down to money?
  • If the ROI is calculated, wonโ€™t that become the basis for the programโ€™s future?
  • Is a negative ROI a death sentence for my program?
  • I work in a nonprofit. Is ROI even appropriate?
  1. Does everything come down to money?

The short answer is no. Sometimes behavior change is the ultimate purpose of a program or process. If so, your evaluation can stop there.ย  The same is true with impact. Not all impact measures need to reflect monetary gains or savings. Impact without the money represents intangible measures in the value chain, but not necessarily less important than ROI. For example, a large technology company had a crash course to teach employees how to work remotely when COVID-19 hit the USA. Ensuring that employees knew how to work remotely was necessary, and a crash course was offered. The executives who requested the course wanted to make sure the remote working solution was effective. They didn’t need to see the ROI. They wanted to know if individuals were working as a team (teamwork), having good coordination and communication (collaboration), and getting their work done properly (productivity).

The team tracked the reaction to working remotely, their knowledge of processes and ability to best work remotely, and their success with working remotely in the organization (Levels 1, 2, and 3). To understand the impact, they asked employees the extent to which this program helped them improve teamwork, collaboration, and productivity using a five-point scale. Data were collected from employees and their managers. Executives wanted the managers’ perspective to ensure that teamwork and collaboration were successful and that the employees were productive. This approach provided executives with data showing the value of this program. There was no need to push the evaluation to the ROI level, because Level 4, Intangible Impact, was all they needed.

  1. If the ROI is calculated, won’t that become the basis for the program’s future?

The answer is no.ย  While there are five levels of outcome data, these levels include six types of data โ€“ reaction, learning, application, tangible impact, intangibles, and ROI. They are all important data sets, and ROI is only one of them. For example, a religiously affiliated healthcare system was experiencing too many complaints of sexual harassment. It appeared that some physicians and leaders were making comments that employees (particularly the nursing team) considered offensive. The complaints of sexual harassment were made to the HR director, who, by policy, had to investigate the complaints and resolve the issue. The root cause of most of the complaints was a lack of knowledge of what constitutes sexual harassment in the organization. Essentially, sexual harassment is in the eye of the beholder. If a person finds another personโ€™s actions or comments offensive, then it constitutes harassment. The organization conducted a learning program on sexual harassment awareness designed to prevent these complaints.1

Reaction data to the program were collected. A quiz about what constitutes sexual harassment and how the policy was administered to measure learning. An anonymous survey to a 25% sample of employees was conducted to measure application. The survey revealed that, for the most part, the sexual harassment activities ceased to exist. Consequently, the sexual harassment complaints declined dramatically, showing a significant cost avoidance. When the cost avoidance is compared to the cost of the program, the ROI was calculated, and the program yielded a very positive ROI.

Was the ROI necessary? No. The purpose of the program was to change the harassing behaviors. Data from the anonymous survey provided evidence that the goal was achieved. So, why take the evaluation to ROI?

The HR director wanted to remind the executives that preventive programs can work. Essentially, the sexual harassment concern was a compliance issue. Individuals need to comply with all the rules and regulations regarding sexual harassment. The HR director wanted to show that tackling this issue was a good business decision. It saved the organization costs, and when you compare the cost of the program to the cost savings, it was a good business decision. In this case, the measure of most importance was Level 3 Application. ROI added the finish touch, or as some people say, it was the icing on the cake.

The main point is that ROI is just one of six types of outcome data and should not be the sole basis for the program’s future.

  1. Is a negative ROI a death sentence for my program?

The short answer is no. A program would only be discontinued if it’s the wrong solution. A negative ROI is not necessarily an indicator a program is the wrong solution. Negative ROIs often merely indicate a breakdown occurred along the way. Programs can break at any point from reaction to learning to application, or even impactโ€”if the program isn’t clearly aligned to the business in the beginning.

Kaycee Buckley, global commercial training manager for a large healthcare company, conducted an ROI study on a coaching program provided to sales managers who coached the sales team.2ย  After conducting the study, she realized the coaching was not as successful as anticipated or desired; therefore, the ROI was negative. When she presented the results to the management group, she included data showing where the program broke down. Essentially, the sales managers were not following through with the coaching. The sales managers realized the coaching was important. They just were not doing their part to make coaching successful.

This study showed clearly that the program was costing more than it was benefiting the company because sales managers were not using what they were learning. The organization changed the approach to solution implementation, and the managers took a renewed effort to make it work and deliver a positive ROI on the next offering. While the ROI was negative, ย the executive group was impressed with the kind of data that the L&D team provided to them. It helped them see completely what was happening.

This type of problem often surfaces when you wait for executives to ask for the ROI. Some L&D managers say, โ€œWe don’t have the request for ROI, so we are not pursuing it.โ€ That’s a risky mindset. When a request for ROI comes your way, three things happen. ย First, you have a short timeline to provide data when you often need much more time to do this. Second, you are now defending the investment decision after the fact. Third, ROI is now on an executive’s agenda.

The key is to be proactive. Play offense and position ROI on your agenda. This positions you to drive the initiative on your timeline. When you discover a program is not working, as Kaycee did, show what needs to change to make it work. This approach puts you in a much better position to avoid the ill effects of a negative study.

  1. I work in a nonprofit. Is ROI even appropriate?

ย Yes, ROI is appropriate. Consider this situation. Janet Simmons is CEO of HOPE Ministries in Baton Rouge, Louisiana. Among other services, her company provides training and coaching to organizations, helping them recruit and train unemployed or underemployed individuals. She’s helping serve a vital segment in the community.

A major foundation is funding some of her efforts to help companies reach out and hire the “unemployable,” as these individuals are sometimes labeled. The companies that work with HOPE Ministries are interested in the services provided and the individuals who need help. At the same time, they invest money and resources into the process. These companies would like to see the impact and even ROI of these efforts. HOPE Ministries wants to see the ROI because they can use this data to convince other organizations to become involved. Both the client companies and HOPE Ministries want to see the ROI of helping these individuals. The foundation doesn’t need the ROI. They just want to make sure these individuals are employed, retained, and are successful in the organization. That’s Level 4, Impact.

We need to think about different perspectives. In this case, the client companies and the nonprofit need the ROI. The foundation funding the ministry doesn’t. ROI depends on the stakeholder’s perspective.

Evaluation studies can be very helpful for nonprofits. At present, Janet is measuring the success of a particular training and coaching program to help an organization with a retention issue, as their turnover is far too high. Janet is measuring this program through the value chain, including the ROI, and using the ROI Methodology as shown in Figure 2. This system is a way of designing for, and delivering, success, and measuring the success of the program.

Figure 2. The ROI Methodology Process Model

ROI is an important measure. It answers the question: Is the program worth it? When to use it depends on your perspective and the purpose of your program.

For information on how this model works or copies of the case studies mentioned (or other case studies), contact ROI Institute or email info@roiinstitute.net.

Reference:ย 

  1. Phillips, Patricia Pulliam, and Jack J. Phillips. Value for Money: Measuring the Return on Non-Capital Investments, ROI Case Studies (Chapter 1: Measuring the ROI in Sexual Harassment Prevention). Birmingham AL, Business Writers Exchange Press, 2018.
  2. Phillips, Patricia Pulliam, Jack J. Phillips, and Rebecca Ray. Proving the Value of Soft Skills: Measuring Impact and Calculating ROI (Chapter 15: Measuring ROI in Coaching for Sales Managers). Alexandria, VA: ATD Press, 2020.