Dr. Robert O. BrinkerhoffOver the past 60 years, many legendary practitioners have made important contributions to the field of training measurement and evaluation. They have shaped our profession and given us the philosophies and tools with which to raise our organizations and ourselves to greater levels of proficiency and effectiveness. In conjunction with the publication of the brand new ASTD Handbook of Measuring and Evaluating Training edited by Dr. Patti Phillips, President and CEO of the ROI Institute, I conducted a series of recorded interviews to capture these thought leaders’ histories, theories, and advice for the profession as well as the creative solutions to the challenges they faced. The commentary contained in these interviews represents the voices of those who have contributed the most in terms of conducting research, applying the methods described in the book, and sharing their knowledge and expertise with others around the world.   Earlier, I wrote about the lessons I learned from this rare opportunity to participate in this unique project and the key messages they shared: http://corpu.com/l/5l/.

This is the final post of an 8-part series, “An Interview With A Legend,” featuring Dr. Robert O. Brinkerhoff, professor emeritus at Western Michigan University and a senior consultant for the Advantage Performance Group. He is an internationally recognized expert in training evaluation and effectiveness and has been a consultant to dozens of major companies and organizations in the United States, South Africa, and Europe. Brinkerhoff is the inventor of the Success Case Evaluation Method, an innovative technique for evaluating learning programs that relies on qualitative feedback to help understand an organization’s training effectiveness.

What follows is an excerpt from our interview (the full text can be found in the “VOICES” section of the ASTD Handbook of Measuring and Evaluating Training now available from ASTD Press).

* * * * *

On how it all started…

I was fortunate to get involved in a graduate program whose focus was program evaluation. I had been five years in the Navy through the Vietnam War period and then after that went to graduate school. It was a very heady time then. I went to the University of Virginia and there was a lot of focus on radical thinking and social action programs and I got very much involved in the social action programs, particularly very radical education programs, street academies and the like. That was also a period where there was a big explosion of thinking and program evaluation. So we had many brilliant theorists who were working at various centers around the United States. That was all a part of the Great Society Program of [President] Johnson’s and evaluation became a federal mandate and requirement so a number of graduate programs started in program evaluation.   I was fortunate to be involved in one of those so I studied evaluation as academic discipline. Much of it was sort of an internship/apprenticeship program so I spent almost five years in Washington, DC as a part of that graduate program working to evaluate some of the major drug education and social action programs of the time.

I was more committed to the goals of the social programs, than I really was to the evaluation, technology and thinking. That started a very early focus on thinking that if the evaluation work wasn’t being used and wasn’t helping people do a better job, then it really wasn’t doing its job. So that was sort of one choice point, realizing that evaluation could be a complete waste of time if all its purpose was was to meet the Federal requirements and put a report in a file cabinet. There was a huge focus on thinking that people had to really use it to do a better job and change things. The other change point came as I happened to get involved in a large evaluation program that was focused on training of special education professionals. We had to think about whether those programs that were training the educators were making a difference to the pupils in the classroom and their families. That sort of thinking, as I began to get more involved in the business and industry evaluation, stayed with me that the real issue was not on figuring out whether the program was working, but rather whether the program was doing any good in a larger sense. I think that very much shaped my thinking about the Success Case Evaluation Method.

We started going to [businesses] as clients and the Success Case Method just came about as a way of trying to be much more efficient about things. So, as we looked at a company that might have put 1,000 people through an executive development program or some sort of management leadership education program, we were trying to find out whether it was working or not. We realized that if we were going to find people using their training to make a difference, that it was much better, rather to choose random samples, it made much more sense to be more purposive about things. We would ask who is the most likely to be using their training and look for them first, because if it is working for anybody, it working for the people who really learned it and were excited about it and felt like they could use it. So, we would, on purpose, select those people beforehand and then follow them up. A very early experience we had was thinking that the best way to be efficient was to look at the level one evaluation data and the level two data; that is, if anyone is likely to be using this training it is probably the trainees who loved the program and who learned the most from it…that is, those who scored well on a end-of-session test. But when we went and followed those people up, what we found was that some of them were using it and some of them weren’t. We had to change our definition of what was success. The best way to find out who is probably using this training is to simply ask them first. We would just do a one item survey of everyone who had been through the training and say, “Are you using your training in a way that you know is making a difference?” Only a small sample of people would say “yes” to that. Then it was those people that we could focus our evaluation inquiry on and just be much more efficient in finding out if the training was making a difference. That really started to shape the way we went about thinking about how and why to do the evaluation.

On how training evaluation has changed over the years…

It comes back to the question of the important issue that people need to recognize is not how are we going to do the measurement but why are we going to do it?  What should we be expecting from this initiative?  What do people need to know about it?  And if we can think clearly first about it, the how questions really solve themselves but if we are not clear about why we need to know and what we need to know, then we will always get wrapped around the axle with figuring out how we are supposed to proceed.  The question I’d like you to ask is: what is the first thing we should be thinking about when we are struggling with evaluation? The answer to that is, why do we need to do it in the first place?

On the progress the profession has made in embracing evaluation…

Let me start with the good news. There are a lot of good people who are thinking about this and doing work with it. You’ve got all of the good work of Jack and Patti Phillips, Josh Bersin, Cal Wick from Fort Hill, Will Thalheimer, Kent Barnett of Knowledge Advisors that are creating technologies that people can use. There is a lot more focus on evaluation and there are more techniques available. These electronic survey methodologies are terrific and make data collection much simpler. There are a lot of good positive changes but things are changing not enough, I don’t think. I have been a presenter for ASTD for almost 30 years. I have gone back and I have had some students do some research on past ASTD national programs and evaluation has been one of the hot topics for the ASTD annual program for more than 40 years. You sort of have to ask: why is this an enduring problem? For 40 years, people have said how are we going to measure this? How are we going to see if this makes a difference? While there has been a lot more techniques available, there has not been that much more clear thinking about why we should be doing it. So, I think there is one thing that has not changed is people may be doing evaluation because they think they have to and not really thinking about why they should be doing it. We are not seeing enough clear, strategic, actionable thinking about why to do it before we start considering how to do it first.

On how evaluation makes a difference in the perception held by executives of training and development…

Many times I have had the luxury of presenting our success case findings to senior leaders and even to the Board of Directors of a company. What I tell them [is], “I have some good news and bad news. The good news is that this $40 million that you have spent over the last three years on this management development training….the good news is that it is working. Let me give you some examples in the ways in which it is working and the difference it is making.”  I will tell them a story, I will say, “Look, here is one of your leaders from an office in the Northeast who used the training and has improved sales by $500 million over the last two years. So that’s training that’s really making a difference…… Now, let me tell you the bad news. The bad news is that it is only happening about 15 percent of the time. In other words, 85 percent of this budget that you’ve been spending in training and development is being wasted. The kinds of changes that I’ve told you are happening are not happening anywhere near enough. Let me ask you, what would your company have stood to have earned if we doubled the number of people who are using the training, as well as these few stories I have told you?”  And then I tell them what difference would that make. You really have their attention when you start telling them about the money and the value they are leaving on the table by not holding people accountable for using the training. What I have found when I present the data in that way and show them that there are a few simple actions they could be taking that by holding their direct reports accountable holding their people accountable for using the training.  And when they begin to see what those simple actions are and they play a key role in doubling, tripling, and quadrupling the ROI from the training they’re getting, they do begin to take action and we have seen it happen many times. Not often enough, but still many times.

* * * * *

Learn more at http://www.advantageperformance.com/businesssolutions/products/advantage_way.html

Listen to the voice of this legend on the ASTD website http://www.astd.org/content/publications/ASTDPress/voices.htm

ASTD Handbook of Measuring and Evaluating Training Telling Training's Story The Success Case Method

Be Sociable, Share!