“If I have seen further [than others], it is by standing upon the shoulders of giants.”

– Sir Isaac Newton

I had the wonderful opportunity to support the brand new ASTD handbook on measurement and evaluation, edited by Dr. Patti Phillips, by conducting a series of interviews and sharing their comments in the “Voices” section of this handbook.   The opportunity to interview these legends of our profession, these giants, was a rare privilege.

Over the past sixty years, many giants have made important contributions to the field of training measurement and evaluation. They have shaped our profession and given us the philosophies and tools with which to raise our organizations and ourselves to greater levels of proficiency and effectiveness.   This work, the ASTD Handbook of Measuring and Evaluating Training, is a unique contribution to the area of evaluation, destined to continue ASTD’s tradition of codifying and disseminating the distilled wisdom of our profession and continually raising the bar.   Its publication could not be more timely.  Few can remember in our professional lives a time when we have experienced such a “perfect storm:” pressure to do more and more with less and less while continuing to delivery quality programs and initiatives that drive business results against the challenges presented by a global economy in crisis.  Without the ability to articulate the impact of our noble efforts in ways that senior leaders understand and value, we are at even greater risk of significant budget and staff reductions.

In a series of recorded interviews conducted over the past few months, these thought leaders shared their histories, theories and advice for the profession as well as the creative solutions to the challenged they faced.  The commentary contained in the transcripts of those interviews, represents the voices of those who have contributed the most in terms of conducting research, applying the methods described in this book, and sharing their knowledge and expertise with others around the world.  To hear the actual interviews, download the podcasts at:  http://www.astd.org/content/publications/ASTDPress/voices.htm

The goal of each interview was to capture, albeit briefly, the history of involvement in evaluation and measurement, guiding philosophies, innovative ideas, tools, models and frameworks as well as predictions for the future.   Generally, these questions formed the nucleus of each interview:

• Tell us about your involvement with training measurement and evaluation – the history of your involvement leading up to what you are doing today?
• How has training evaluation changed over the years?
• What progress has the training industry made with evaluation? Are they embracing it or avoiding it?
• How do executives really view learning and development and the investments made in these processes?
• Has evaluation made a difference in the perception held by executives of training and development?
• Why do we still see such a low investment in training measurement and evaluation within organizations?  How can we facilitate more investment in the future?
• What does the future hold for measurement and evaluation?

The full transcripts (found in the “Voices” section of the handbook) and podcast recordings provide a deeper glimpse into their thinking.

What, then, are the “learnings” from these interviews?  What forecast for the future of training evaluation and measurement can we derive from the comments of these legends?

• Standing on the shoulders of those who came before, the history of measurement and evaluation is evolutionary, not revolutionary.
• Many of the innovations came from examining what had been done and simply asking, “why” and “what if?”
• Don Kirkpatrick, Jack Phillips and Bill Rothwell use the same analogy about proof and evidence before a “jury” which increasingly holds our fate in its hands.
• There was great consensus that there is (and will continue to be) an emphasis on evaluation data at the higher levels, moving beyond reaction and learning (although still important) to behavior change and business/financial impact.
• Dana Robinson articulated the belief that behavior change that drives results is the only real determinate of value.
• We need to make our evaluation work a critical element of the “front-end” work, never as an afterthought.
• There is no lack of tools, methodologies, case studies and now technology to assist us but they are not a substitute for clear thinking about what we are trying to measure and why.  Evaluation is less about the methodology chosen but more about the rationale behind the selection.
• The field of evaluation is not without differing views on key issues.  For example, Bill Rothwell says his executives suggest training professionals not evaluate their own programs; yet, Jack Phillips’ research states that executives want the evaluation data. So what is a training professional to do? They need to approach evaluation methodically, systematically, and ensure they make conservative assumptions. They need quantitative as well as qualitative data along with financial and non-financial data. Augmenting data with quotes, examples and specific case examples from highly-regarded executives and stakeholders can not only help drive home the impact but also these impeccable sources help mitigate executive concerns about objectivity.  This is what gives results credibility and helps senior leaders trust the evaluation data.
• Partnership with and communication to senior leaders is at the heart of long-term success.
• Both Rob Brinkerhoff and Roger Kaufman look to a future when evaluation also encompasses the impact on society and “the greater good.”
• Rothwell’s “Theory of Visible Activity” explains a lot about executive perceptions.
• Dana Robinson’s experience notwithstanding, evaluation work does not always lead to finding your soul mate.
• Among all those interviewed, there was a sense of appreciation for the journey so far, for the many who have embraced the methodologies and practices that shape their evaluation and measurement work but there was also concern for the relatively small percentage of members of our profession who do this along with a frank assessment of how much further we need to go.

There is no way to capture the depth and breadth of the contribution to the profession, in particular to the field of training evaluation and measurement, of these legends…these giants… in a book chapter or in brief interviews.  To be fully appreciated, the richness and complexity of their thought leadership must be experienced through reading their works, studying with them if you have that fortunate option, and then applying the concepts and tools to the work we do for our organizations and clients.  This handbook is an important contribution to the profession, a compilation of what we know about measure and evaluation by the people who have blazed the trail and those who continue the journey.  Perhaps this “Voices” chapter offers a roadmap for your own continued exploration or a blueprint for the next iteration of innovative thinking about evaluation so that, twenty years from now, the next series of interviews with “legends” will include your voice.

It was my privilege to serve as the interviewer.  I have been a beneficiary of their thought leadership and practical frameworks for many years and, if I have been able to effectively serve the clients and corporations for whom I have worked, it was due, in large part, to the pioneering work of these legends.   I remain deeply grateful to Dr. Patti Phillips for the opportunity to be involved with this definitive work and to stand on her shoulders as well for many years to come.

Be Sociable, Share!

•