Thank you for bringing up the topic of criticisms of Kirkpatrick’s model. I did research into the area and found some things I expected, and some things I didn’t. What I expected was a hornet’s nest as the research is overturning the behaviorism assumptions that 20th century training was based on in light of 21st century research in cognitivist theory, the brain and workplace changes due to technology.
And a hornet’s nest it is, with many falling on the side of Kirkpatrick. It seems as though his theory and company have been very influential. Trainees now expect a smile sheet and a post-test. Managers expect some sort of level 3 and level 4 data. And not many want to change this expected model.
Many realize there are logical holes in Kirkpatrick’s system. Ried Bates gives a nice criticism of the flawed logical assumptions in the model that it assumes causal links and incremental levels. Meanwhile, Stephen Gill points out that much of the data gathered by the model is self-reported. Who doesn’t like to rate 2 days off of work learning as better than working?
In response to these criticisms, there is an updated model of the 4 levels. However, this appears to be more of a window dressing with a flow chart outlining the process of the evaluation. It still contains the same logical flaws.
Another unexpected thing was how Kirkpatrick and Associates has become a cult of personality, a creaky old edifice whose humble, apple pie exterior was unwilling to adapt to modern reality. Then I read Kirkpatrick’s original text and saw stubborn, circular logic.
What I didn’t expect to find was the void of another model. There are models, but not as clear as Kirkpatrick’s 1, 2, 3, 4 simplicity. One that leverages this is Kaufman, Keller, and Watkins. They add organizational elements and suggest learning ecosystems, but keep the same basic 4 levels.
Other proto-systems would include applying 6 Sigmas to evaluate training, using a modified narrative theory like Brinkerhoff’s Success Case Method, or a system’s view approach such as Stufflebeam’s CIPP evaluation model.
To make a philosophical leap, the basic problem seems to be The Game of Work and The Game of Training, which is based on the Game of Learning. When we break the 18th century idea of schools, training and organizations that we dwell in, we will have to discover the experience, social, mistake driven cognitive reality that is learning, and how to evaluate it, probably not so simplistically and mechanical.
Yes, we are driving around in 54 Dodge’s with the evaluation methods of Kirkpatrick. But it gives us enough seemingly valid justification for it that our dopey leaders, who don’t really understand learning or training, check off on it. So why change? More appropriate, why break the wheel that feeds us?
Bates, R, 2004, A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. http://goo.gl/Up5P3h
Gill, S, 2011, Kirkpatrick’s Four Levels of Training Evaluation: A Critique. http://stephenjgill.typepad.com/performance_improvement_b/2011/05/kirkpatricks-four-levels-of-training-evaluation-a-critique.html
Kaufman, R, Keller, J., Watkins, R. (2007). What works and what doesn’t: Evaluation beyond Kirkpatrick. http://home.gwu.edu/~rwatkins/articles/whatwork.PDF
Kirkpatrick, J, Kirkpatrick, W.K., 2009. The Kirkpatrick Four Levels: A Fresh Look After 50 Years 1959 – 2009. http://www.kirkpatrickpartners.com/Portals/0/Storage/Kirkpatrick%20Four%20Levels%20white%20paper%20updated%2010%2009.pdf