The idea of AI has existed for years, but it truly gained public attention in November 2022 when ChatGPT was launched. For the first time, individuals could see the power of what is known as generative AI. What it can do is amazing, and the reaction to it everywhere has been breathtaking. Investments in AI are skyrocketing, and organizations developing AI have mushroomed. Essentially, AI is now everywhere. Leaders love it, consumers demand it, investors reward it, and many outlets promote it.

AI is the part of innovation where computers are involved, and functions that are normally performed by humans are now automated. AI is incredibly popular, but it does have some issues. One is the concern that many AI programs and projects fail to deliver the desired results. While our initial reaction to AI was “Wow! This is amazing! Let’s get involved because we don’t want to get left behind.” Unfortunately, the more we learn about AI, the more we learn about some of its deficiencies.

For example, we asked ChatGPT a simple question: Describe the ROI Methodology. Although the process is incredibly well documented in 75 books and 300 articles and has thousands of users globally, we were surprised at how inaccurate the ChatGPT answer was. In fact, some of it was absolutely false. We know we need to train AI, but we were concerned about the people who didn’t know the real answer, and that includes most people. This illustrates the downside of AI, as there are many flaws and weaknesses.

AI does hold great promise, but it didn’t take long until the reaction to AI moderated somewhat as people realized there was some risk. We need to establish guardrails on AI as it can be misused, inaccurate, harmful, and wasteful. According to Warren Buffett at his 2025 shareholders meeting, “We will regret that we invested so much in AI.

Where’s the AI-driven utopia we were promised? It may not be here yet, but with the right approach, it’s within reach. One example comes from BMW, where the IT team explored using AI for automatic software testing. Testing is critical whenever software is developed or updated. If it’s skipped or done poorly, the results can be disastrous, as we saw in 2024 when a software flaw from CrowdStrike caused major disruptions for banks, airlines, and other organizations. The promise of AI, in this case, was straightforward: faster, cheaper, and better testing.

BMW took a thoughtful, data-driven approach. Instead of simply trusting the promise, they asked ROI Institute to prove it. Over two days, the team carefully evaluated whether automatic testing could match or outperform manual testers. The results were positive. Importantly, it delivered quality that matched, or even exceeded, the manual efforts by avoiding common human errors. Also, AI was faster and likely cheaper, as it could replace the work of 32 manual testers.

But BMW didn’t stop at cost savings. They focused on how AI would affect their people. Recognizing that software testing jobs had high turnover and weren’t satisfying roles, they worked on plans to manage staff transitions through attrition and reassignment. This thoughtful strategy shows how AI’s return on investment goes beyond speed and cost. It’s also about delivering value while respecting employees and strengthening the organization’s culture.

That’s the way it should work. To get there, we have to be prepared to evaluate the success of an AI application. An article in MIT Sloan Management Review by Thomas H. Davenport and Randy Bean highlights important trends with AI. The top two trends in AI data science are

1. Leaders who grapple with both the promise and hype around agentic AI, and
2. The time has come for measuring results from generative AI experiments.¹

So, how exactly do we do we measure the success of AI? Let’s explore the Chain of Value for AI solutions.

The AI Value Chain

Figure 1 shows that for any AI application, people are involved. The input into the process includes volume (the number of people involved), their hours of involvement, and the cost of the AI platform, subscription, tool, or process. It also includes the convenience of the solution.

The Outcomes

There are five levels of outcomes that enable us to see the value of AI unfold.

Level 1, Reaction, is how participants see the particular AI tool or process. Is it relevant to their work? Is it important to the users? Is it something they intend to make successful? Would they recommend it to others? These are the top four reactions that need a resounding yes for AI to be successful. Reaction will predict the application, which is level 3 in this process.

Level 2, Learning, is learning how to use the AI tool or process.

Level 3, Application, is using it properly, using it often, and having success with it. At this point, things sometimes break down because they are not used properly, so you have to look at the barriers and enablers—what kept them from being successful with it and what helped them.

Level 4, Impact, includes measures such as productivity, time, quality of work, cost savings, image, reputation, engagement, and compliance. Impact is going to be in two categories: the measures that you can convert to money that will go in the ROI calculation, and the measures that you cannot convert to money credibly, the intangibles. Those impact measures that are converted to money are used in the calculation as a monetary benefit. But first, you can see that there must be a step to isolate the effects of the AI on the measure. If productivity was improved, it could be that there are other influences driving these measures. You must sort out the effects of AI.

Level 5, ROI, is expressed in two different ways: as a benefit-cost ratio (BCR), which is the monetary benefits divided by project costs, or ROI, expressed as benefits minus costs divided by costs times 100. ROI is the ultimate accountability.

Figure 1. The AI Value Chain.

This value chain is probably very familiar. It was based on a logic model that dates back to the 1800s, and it has been used in all types of fields. It is the same logic model used in many publications from ROI Institute as shown in Figure 2.

Figure 2. ROI Institute Publications that Reference the Chain of Value.

Hard and Soft Data

It is helpful to think about the impact measures in four major categories for what we often call hard data: output, quality, cost, and time. These four categories of hard data items are present in any organization, regardless of its type. There is output (the work that is being done), quality (the mistakes), time (to perform tasks), and costs (the cost of doing the work). These measures are hard data because it is relatively easy to assign a monetary value to them.

AI should affect all these measures, and it depends on the organization. The innovation for university outcomes will be different from a hospital’s outcomes, and those will be different from a manufacturing plant’s outcomes. These categories are present in every organization.

Many soft data items are more difficult to convert to money. They are often connected to these hard data categories, but often fall into the soft data category. These include measures such as employee development and advancement, client service, image, reputation, client satisfaction, leadership, and innovation. Assigning monetary value to these measures is more difficult.

Figure 3 shows examples of AI’s many intangible measures. The intangible measures are just as important, if not more important, than the tangibles that go into the ROI calculation. Essentially, you are saying AI should be connected to these measures through this value chain, but it is often not.

Figure 3. Examples of Intangible Measures Linked to AI.

Does AI Deliver?

In the last two years, the issue is not whether you have AI technology that is going to replace people, but how to use generative AI as a tool. The ROI for apps such as ChatGPT+, Claude Pro, Gemini Advanced, Perplexity Pro, and Character AI will likely involve time savings, productivity, quality, cost, and intangibles.

We would offer this recommendation: If you are purchasing a platform, tool, or subscription that is costing you money, ask your provider, “Do you have ROI data for the application of your services?” If they say yes, ask to see this data. If not, ask why not, and could you? Require it from them, demand that they do it. If enough people do, they will do it.

In a recent keynote presentation to 4,000 individuals, I asked the audience, “How many of you have actually seen an ROI study on anything with AI?” Out of 4,000 people in the audience, not one person raised their hand. That’s the problem. It’s not being done, yet it needs to be done. So, how do you do it?

The ROI Methodology

More studies need to be conducted on the value of AI, particularly at the ROI level. However, all five levels of outcomes are needed in the study to show you clearly how the impact and ROI are delivered. This data is needed for those already supporting AI, as they need convincing data to continue to support this. For those who are curious, this will remove some of the mystery about the value. For skeptics, this helps convince them that AI is or will deliver results. For critics, an evaluation may address some of their concerns.

It’s helpful to remember that not every project should be subjected to this type of analysis. If you are paying $20 a month for a subscription to a generative AI tool, an ROI study is probably not needed. However, if you are paying $200 a month and it involves 10,000 individuals, you need an ROI study. Or, if an AI platform is very expensive upfront, it needs this evaluation process. In fact, we suggest that only 10 to 20 percent of AI projects really need analysis at the impact level, and maybe half of those should be pushed to ROI.

The selection criteria for those projects begin with the cost. The more expensive it is, the more you need it. Also, the importance of the issue being addressed. Is this a vital metric or series of metrics in the organization? Is AI connected to your strategy? If you know you have to become more efficient in a particular area or are you having difficulty getting people to staff for a particular area, AI might be the tool to prevent that. Often, a part of the strategy is to reduce the number of employees through additional investments in AI. Finally, some projects attract attention from senior leaders, and it is usually the previous three areas that attract this attention. They often ask, “So what’s the ROI of all of this?”

Now let’s get started. Figure 4 shows the ROI Methodology model. It is interesting that this model is not just an evaluation system, although it does evaluate the process. (In fact, it is the most used evaluation system in the world.) It is also a system that actually designs for the results because it requires the end-goal as you start any process. Finally, it helps you deliver results for the stakeholders involved in the project. All the stakeholders play a part in making sure it delivers the results that you identified in the beginning. Let’s examine each step.

Figure 4. The ROI Methodology Process Model.

Start with Why

Every AI tool should have an end goal, driving a particular business measure. Those business measures will be in the output, quality, cost, or time categories, and perhaps other metrics. If you are a hospital, you might be interested in the length of stay. If you are a university, it could be the graduation rate. If you are a retail store, it could be sales growth. If you are a corrections department, it could be recidivism. Start with that end in mind clearly. The end goal is not doing something different but the impact of doing something different. It’s not the experience of AI, but it is the impact of AI.

Make it Feasible

This focuses on ensuring AI is the right way to move the measure. If it is to increase sales, is this the right way to do this? If it is to improve the quality of services or reduce complaints, is this the right way to do it? The solution is often inherent in the system by identifying a problem and understanding what is causing the problem. Sometimes, the solution involves automating a process and ensuring that your automation (AI) will actually improve the measure.

Expect Success

This step involves three issues.

1. The impacts are clearly defined and communicated to the entire team. They become the reason you are doing it. You are not successful until you have an impact. Success doesn’t occur when you have a tool that is impressive and easy to use. It occurs when you have the consequence of using it, and that’s the impact.
2. Set objectives of how people involved in the process react to the AI, learn to use it properly, and use it successfully. Then there is the impact that comes from it. Objectives need to be set for reaction, learning, application, impact, and yes, ROI, if you are evaluating ROI. These objectives define the minimum acceptable performance and clearly describe how individuals should react, what they should know, what they should do, the impact that will occur, and the minimum return on investment that is necessary for you to acquire it.
3. These objectives, which show the roadmap to success, are provided to everyone on the team to make sure that they know what they must do to make this success occur.

Collect Data, Make it Matter, and Make it Stick

Data collection is in two major categories. The first is capturing reaction and learning data. Make sure that the target individuals for AI see the value in it and have learned to use it properly. This is important because the learning often affects reaction, and the reaction inspires more learning. It’s reciprocal. A project can easily break down at this point if it is rejected before participants do or see anything, or they don’t know how to use it properly.

The second category of data collection is application and impact, making sure that participants are using it properly and tracking the impact that is coming through it. These impacts are the same measures that are identified in Step 1. At this point, for impact, the only evidence is that you made a difference because the impact measures have improved. You need to change this to proof, and that leads us to the next step.

Isolate the Effects of the Project

While the AI solution is often improving the impact measure, there are often other influences around that same measure. Think about sales, for example, there may be a dozen influences trying to drive sales. For credibility, it is important to sort out the effects of the AI from other influences. There are many methods and techniques to do this, ranging from experimental versus control group to trend line analysis, and estimates from the most credible source. Sometimes, the users of AI are best positioned to report its results.

For example, using a generative AI tool saves a person time. That individual is the best person to sort out how much time it saved them. Because that is an estimate, one of the ROI Methodology standards will come into play. Guiding Principle 7, which states that estimates of improvement should be adjusted for the error of the estimate. You do that with a confidence adjustment. If a participant reports that ChatGPT saved them ten hours per week. You will follow up and ask what their confidence is in that number on a scale of 0 to 100 percent. If they suggest 80 percent confidence, then they admit that there is a 20 percent error in that number. After all, 100 percent is certainty. Therefore, 20 percent of that estimate must be removed. To accomplish this, multiply the estimate by the confidence number. Ten hours multiplied by 80 percent leaves eight hours, so you claim the eight hours. This is very important as estimates become the default method for isolation. It can always be done, and it is being done in every evaluation.

Convert Data to Money

You make it credible by having a credible monetary value for the measure. A standard value means that the number has already been calculated, approved by management, and reported to the people who need to know it. As reported by ROI Methodology users, 52 percent of measures use standard values. This is great news. If not, there are experts. Forty-five percent of measures are converted using experts or individuals who work with the metric. They collect it, report it or are concerned about it, and sometimes have an entire team concerned about it. They often know the cost of something or the value of something, such as the lifetime value of a new customer. There are also external data sources and estimates. The point is that the monetary value can be determined.

Identify Intangibles

If a measure cannot be converted to money credibly within a reasonable amount of time, it is left as an intangible. It is still important, but just not converted to money. These are measures such as image, reputation, teamwork, collaboration, and brand—measures that may not be converted to money, but they are still important. The key is to identify these measures and have the people who are using the AI tell you the extent to which this solution is actually influencing that measure. That is an easy way to connect the AI solution to the intangibles. Intangibles are important from the perspective of executives who fund all types of projects based on these measures.

Capture the Costs

All the costs for the AI solution are detailed. This includes not only the cost from the provider but also the cost of the people involved in it, whether internally involved with implementing the process or supporting it, or those who have taken the time to learn it. Any cost that is involved is included, and remember, time is a cost. Include all costs, even the cost for the evaluation, to fully load the costs and make the calculation credible.

Calculate ROI

The ROI calculation involves the BCR benefits divided by costs. This is a measure that has been around for centuries. It comes from cost-benefit analysis that had its origins in governments many years ago. The ROI calculation and BCR are shown in formula form in Figure 5. ROI has been the dominant way businesses manage their organizations for the last 100 years. But in the last two decades, it has been an important number for the non-business groups as well. Collectively, these are the two most common ROI metrics on the planet.

Figure 5. ROI and BCR Calculations

Tell the Story

Stories are identified through comments collected when the reaction, learning, application, and impact data are collected. The comments tell the stories of the people who are making it work and having great success, as well as the people who are struggling and finding it difficult. Stories bring the evaluation to life, but the numbers are needed as well. You need to have both numbers and a narrative. You use the stories for interest, engagement, and connection, and the data is for validation, verification, and decision-making. You need both. This provides a balanced set of data to make decisions, which leads to optimizing results.

Optimize and Leverage Results

The evaluation results need to be used for good going forward. It could be used for process improvement or to gain support or commitment. Today, the most important reason is funding. ROI evaluation is used to secure funding for the project or to keep the funding going for a project.

AI providers should use this data to market their services to others. Money is always tight, and this type of evaluation can provide the data needed to make that funding decision and keep supporting the project.

So there you have it, a 12-step process of evaluating an AI solution.

Summary

We recommend that AI providers be proactive. You owe it to your customers and potential customers to show the value up to and including ROI. If you are purchasing or acquiring an AI solution, you owe it to your organization to show the value. Don’t wait for someone else to ask you for it. If you wait for someone to ask about ROI after the solution has been fully implemented, it may be more difficult to capture the data and make this analysis. It will take time. Three things happen to you:

1. You will have a short timeline that you won’t be able to meet.
2. You are now in a defensive position; you want to be on the offense with this.
3. You now have ROI on someone else’s agenda. Keep it on your agenda.

As Patti P. Phillips, Ph.D., CEO of ROI Institute, often says, “Remember, when it comes to delivering results, hope is not a strategy, luck is not a factor, and doing nothing is not an option. Change is inevitable; progress is optional. It is really up to you to make it work, and you can.”

For additional information, please contact ROI Institute at info@roiinsitute.net.

This article was originally published on August 1, 2025, in Spark Magazine.

References

1. Thomas H. Davenport and Randy Bean. Five Trends in AI and Data Science for 2025. MIT Sloan Management Review. January 8, 2025. https://sloanreview.mit.edu/article/five-trends-in-ai-and-data-science-for-2025/