What Does the New ISO Standard Mean for L&D Measurement?

Dave Vance, Ph.D., author ofย The Business of Learning: How to Manage Corporate Training to Improve Your Bottom Line, Second Edition and co-author of ย Measurement Demystified: Creating Your Measurement, Analytics, and Reporting Strategy andย Measurement Demystified Field Guide.

In June 2023, the International Organization for Standardization published its first comprehensive standard for learning and development metrics.

The standard, ISO/TS 30437:2023: Human resource managementโ€”Learning and development metrics, has the potential to significantly improve L&D measurement and advance the maturity of our profession.

Need for the standard

Practitioners continue to ask what should be measured, how it should be measured, and what should be done with the metrics once they are calculated. For some metrics,ย we donโ€™t even agree on what to call them or how to define and calculate them. This is not surprising since there has not been a comprehensive framework, common language or agreed-upon definitions for measurement and reporting.

We have the Katzell/Kirkpatrick/Phillips five levels for program evaluation, but this accounts for only five to 10 of the 100-plus metrics we have in learning. How about the rest, including metrics for use at the department level (for example, percentage of on-time completions and reach) and for organized informal learning (for example, performance support), which is becoming increasingly popular?

Contrast the state of L&D measurement in particular and our profession in general to that of more mature professions such as accounting, chemistry, or biology. These mature professions have agreed-upon frameworks and a common language that facilitates understanding and communication among practitioners, as well as the development of standardized curricula at universities and conferences. For example, accounting has four types of measures (income, expense, assets, and liabilities) and three basic statements (income, balance sheet, and cash flow). Accounting terms are well defined, and accountants agree on how to calculate and use their measures. A comprehensive framework and common language are the foundation for any profession.

Standard names and definitions are also important if a profession is to mature. Imagine if accountants did not agree on how to calculate the values of the measures in the financial statements. Imagine if scientists did not agree on the properties of the elements in the periodic table. It is hard to advance as a profession when there is no agreement on terminology.

Standard names and definitions are also required for accurate benchmarking. Without agreement on how to define and calculate metrics, organizations submit data for โ€œtheir versionโ€ of a metric, which means that the resulting aggregated results are not as accurate as they could be if all reporting organizations calculated their data uniformly.

In short, there has been an unmet need in our profession for an agreed-upon framework and common language.

A deeper look at the standard

Developed by a global team of ISO industry experts from 2021 to 2023 (who I had the opportunity to lead), this 42-page standard provides a framework and guidance for measurement and reporting. The standard was created to provide clear, practical advice to practitioners on how to create a measurement and reporting strategy as well as how to define and calculate the metrics. It begins by describing the five broad types of data users and the four broad reasons to measure. Next, the standard shares three categories of metricsย and four types of reports.

The standard recommends 19 metrics for smaller organizations and 52 metrics for larger organizations, including metrics for informal learning (knowledge sharing through communities of practice, performance support, and non-course-related content available through an organizationโ€™s learning portal). Recommendations are made for each type of user, which makes it easy for the practitioner to select the most appropriate metrics for each user. For example, eight L&D metrics are recommended for the CEO of a smaller organization, 13 for the head of learning, and seven for a program manager. The standard also includes recommended names and definitions for each metric and examples of their calculation and use in scorecards.

The framework, based in part on TDRP (Talent Development Reporting Principles, created between 2010 and 2012 by a group of industry thought leaders and practitioners), is designed to be easy to understand, easy to use, and easy to remember, much like the Katzell/Kirkpatrick/Phillips framework which employs just five levels. A framework with more than four or five categories may be more accurate and discerning but is likely to be harder to remember and use.

With usability in mind, the ISO standard includes just five broad categories of users: Senior organization leader (for example, the CEO), group leader (for example, head of a business unit), head of learning (the CLO or person in charge of learning), program manager (responsible for developing and managing the program) and individual learner. Guidance is to begin by knowing your users since different users will be interested in different metrics.

Likewise, while there are many reasons to measure, the standard collapses them into four broad categories, which also dictate the type of report to use when sharing the data. The most common reason to measure is to inform (answer questions and identify trends). The second reason to measure is to monitor (ensure the reported value of a metric is at least as good as it has been historically). The third reason to measure is to evaluate a program (determine its effectiveness and impact). Finally, the fourth reason to measure is to manage a program to deliver planned results (requires specific, measurable goals and special reports generated monthly to determine if additional actions are required for success).

Three categories of metrics are offered (efficiency, effectiveness, and outcome). Efficiency metrics comprise the bulk of L&D metrics and include activity, utilization, and cost metrics). Common examples include the number of participants, completion rates, and cost. Effectiveness metrics include participant reaction, goal owner satisfaction, learning, application, and ROI (return on investment). Following TDRP and a number of thought leaders, the standard separates outcome metrics Level 4) from the rest of the effectiveness metrics (Levels 1-3, 5) since the isolated impact of learning (Level 4) is the metric most CEOs want to see but are least likely to receive from their L&D department.

The ISO framework concludes with reports. Scorecards and dashboards are the most common. Scorecards typically are Excel spreadsheets with metrics as rows and time periods as columns. While scorecards are excellent for presenting detailed data, dashboards are designed to be more visually appealing with charts and graphs. Dashboards typically show less data, but it is more aggregated (for example, year-to-date). Program evaluation reports are designed to brief the results of a program upon completion. Management reports are the fourth type. These are specially designed reports to use monthly when managing a program to a successful conclusion.

With a common framework and language established, the standard recommends the following:

  • Know your users and learn why they want the data. In other words, what is their reason to measure? Employ the four-reason framework as your mental model.
  • Recommend appropriate metrics from the standard to meet their need. Once the type of user is known, simply refer to the table in the standard for the list of recommended metrics. There should always be several efficiency and several effectiveness metrics. Include an outcome metric if the program is designed to help achieve a specific, measurable organizational goal (for example, increase sales by 10 percent). Use the standard name of the metric.
  • Calculate the value of the selected metrics using the definitions and formulas in the standard.
  • Share the data with the user in the recommended report, which is determined by the reason to measure:
    • Inform: Scorecard or dashboard.
    • Monitor: Scorecard or dashboard with a threshold or legend based on an historical best.
    • Evaluate: Program evaluation report.
    • Manage: Management report.

This approach applies to creating measurement and reporting strategies for both individual programs and for the department as a whole.

A framework for the future

The new ISO standard is designed to address the need described above for a framework and common language as well as for standard names and definitions. It also provides detailed, practical guidance for selecting metrics for both smaller and larger organizations, including metrics for both individual programs and for the department as a whole as well as for informal and formal learning.
This standard should help our profession become better organized and more mature, which in turn will allow us to have greater impact and become better business partners, but only if it is widely adopted and used. So, please considerย purchasing a copyย and implementing in your L&D group. Even if you do not buy the standard, you can begin to use the framework for types of users, reasons to measure, types of metrics and types of reports.

 

This article was originally published on August 9, 2024 at ChiefLearningOfficer.com

763 722 769 765 761 759 448 444 442 440 438 430