John R. Mattox II, Ph.D., principal consultant with Metrics that Matter, a Division of Explorance, recently spoke to a group of Atlanta-area learning leaders about how to measure learning programs for impact and the new technologies that are available to help with the measurement process.
“Training programs should have a measurable impact on learning, application, performance and business outcomes,” Mattox said. “There are many ways to measure these outcomes, but the two most effective ways are impact studies and comparison to standards.”
Impact studies, for example, involve randomly assigning a large group of employees to two groups: a test group that would receive training and a control group that would not. Other than the training, the groups would be treated exactly the same. After the training, business outcome (e.g., team productivity) would be measured for both groups before, during, and after the training event.
A comparison-to-standards approach involves gathering large amounts of qualitative and quantitative data are gathered through surveys and benchmarks are created. Then the results are compared to internal and industry benchmarks. With that information, actionable reports are created for various stakeholders via executive dashboards, management reports, and tactical reports.
A new learning ecosystem is emerging with tools to help measure learning effectiveness. Here are some elements of this brave new world.
Adaptive Testing – Customizing learning for the learner
Through adaptive testing, Educational and psychological testing processes determine a learner’s capability and presents customized content based on the knowledge and skills of the learner. Learners pass a test on a content domain instead attending a class and then taking a test.
Learning Record Store – Tracking where learners go to learn
A learning record store captures employees’ formal and informal working and learning activities through the Web sites and apps an employee uses during the work day and provides insight into learning outside of the standard formal curriculum.
Natural Language Processing: Evaluating what is said and what is meant
Natural language processing involves directing computers to recognize meaning in written and spoken words using analysis and comparison. For example, by using NLP to summarize comments provided during training evaluations, NLP can determine whether a learner is being sincere or sarcastic whey they write, “Loved the instructor!” on an evaluation.
Machine Learning: Predicting behaviors and successes
By processing data and detecting patterns, machine learning can be used to predict who will be successful in a class or a curriculum. It can also suggest additional classes for a learner who needs more work in an area and recommend that an advanced learner skip classes.
As learning leaders consider where to make investments for the future, they need to consider whether to invest in new technology, additional personnel, solid processes, or other areas. “Are you investing in volume or value? Mattox asked.
Mattox closed his presentation with a quote from Dave Vance, Ph.D., executive director, Center for Talent Reporting and former CLO at Caterpillar:
“If I had an extra $50 -$100,000 to improve my company’s learning function, I would not spend it on AR / VR, AI or the next shiny thing. I would spend it on the tools and processes that would close the gaps in fundamental operations. Am I measuring, monitoring, and managing the right things? That comes first.”
You can learn more about John Mattox on LinkedIn.
