Building Measurement into Our Training-Development Process

Another lunchtime webinosh with Dr. Will Thalheimer. The topic today: Building Measurement into Our Training-Development Process.

Guest speaker today is Roy Pollock, CLO of Fort Hill Company. Roy wrote Six Disciplines of Breakthrough Learning. Roy has a new book: Getting Your Money's Worth From Training & Development. A Guide to Breakthrough Learning for Managers/Participants. (A manager can make or break a training investments; must get managers involved in a training initiative to get the maximum return.)

Roy and Will have been teaching a one-day workshop on measurement. Roy takes business line; Will takes the professional learning side.

Learning News

• Microsoft just released Semblio, a new authoring tool
• Skype 4.0 (much better video)

Question of the Week -- Economy

How has economy affected your Learning Unit?

• Hit Us Hard 31%
• Hurt A Little 33%
• Not Much Effect 33%
• Helped Us 3%
• (Is "it hasn't hurt us yet, but I'm nervous" an answer?)

Measuring Learning

3 Reasons to Measure Learning

1. Prove Benefits
2. Support Learning (testing is a useful learning strategy that helps people retain info)
3. Improve Design

Outcomes Planning Wheel (Roy's model)

1. What business need(s) will be met?
2. What will participants do different and better?
3. How can we confirm these changes?
4. What are the measures of success?

Will's Learning Landscape

1. Learning Intervention (Learner Learns)
2. Performance Situation (Learner Retrieves/Learner Responds/Learner Applies
3. Learning Outcomes (Learner Fulfillment, Learning Results)

Usually do Level 1 eval as during Learning Intervention

During Performance Situation -- Learner Applies -- could delay the L1 smile sheet. could do a Level 3 support at that point.

Level 4 Eval traditionally done during Learning Outcomes

(there were a lot more things on his chart, but I'm not that fast and it was somewhat complicated...)

Most of us are doing Level 1 completion and smile sheets. Some are doing Level 2 recall or decision making. Fewer doing Level 3 and 4.

Roy: "If I'm a business leader, I really need to know if training is working in order to invest $$."

The Job Aid: Building Measurement Into Your Training-Development Plan

1. Identifying training opportunities (drive performance by improving knowledge/skills). This can be done by learning and/or business leaders. L&D can be proactive and add real value.

2. Underlying business needs are clearly articulated. Let's get clear about business needs. Training is an investments, it must serve business needs.

3. What will participants do better and differently?

4. Is training the right solution? Often training is the hammer to every nail. Not every business challenge can be solved through training.

5. Besides training, what else is required to produce the desired behavior? Training is rarely the whole solution. What other support mechanisms, systems, etc. need to be put in place? Business managers need to be part of solution in order to ensure that training sticks and that performance does improve.

6. What are the relevant metrics? Match to business and learning imperatives. Take time to define the measurement and define the outcome -- must be done up front. If you define success up front, it makes it easier to be successful in your design. Business leaders want to be changing performance on the job.

7. Get sign-off from all stakeholders on behavior change goals, resourcing responsibilities, metrics. Talk with stakeholders every time. Don't just do a cookie cutter design.

8. Design and develop the training and follow-through. Unless learning is taken back on the job and practiced, then it won't produce results. It's all about learning transfer which requires follow-through.

9. Make measurement part of design and development. Must be built in.

10. Pilot test training prototype and improve it.

11. Pilot test measurement instruments and improve them.

12. Deploy training; support on-the-job application.

13. Deploy measurement. Collect data.

14. Analyze data. Report results. Take action. How can we use this data to sell our story (the story of L&D) to assure continued funding.

15. Make improvements. Plan future improvements. In a six sigma cycle.

A philosophy of measurement in one page.

You can download this job aid and others on Will's site.

Questions

Question asked about measuring results social media and wikis. Roy talked about getting examples from learners and anecdotes.

When training a new system (IT, software) how do you separate the benefits of the system with the benefits of the training? Understand what the training is supposed to measure...What can training itself produce as part of the process?

Using a blog to self-report application of something new. What's better - the learner's report or the manager's report? It depends. Sometimes the manager has no idea what the learner is doing. Sometimes the manager does know. Don't just limit to asking managers. If program is about improving customer service -- then ask customers. Who would observe this? Who would be in the best position to observe a change? Could be people. Could be a system.

You could go all out and separate the effects of training by using control groups. Give some people training and others no training. etc. But the level of measurement has gotta be proportional to the strategic value of the course and its benefits.

How do you quantify self-reports when there's no business metric? Always the danger with self-reports that the learner exaggerates. Use Brinkerhoff's Success Case Measure: If you got a result, describe it.

What are the simplest, most effective tool to measure learning? The # 1 tool was Captivate (in eLearning Guild Report on Measurement). Will says the #1 tool we should be using is our brains. It's not about the tools.