What are some of the benefits and pitfalls of an assessment plan?

There are many benefits to having an evaluation plan. In fact, common sense says it is essential. How can one justify that a training has worked without evaluation?

Here are some ways evaluation plans help. At the most basic level, it helps evaluate the instruction. This may seem oversimplified, but there are still many trainings that give a 10 question quiz at the end that is used to measure if the instruction was effective. I recently took a safety training with many details. Being able to pass the quiz and remembering the data a week later are not really related. How do they know I can apply that knowledge? Finally, how do they know if I have a safety attitude? Not having an evaluation plan can lead to wasteful instruction with little usefulness because it is based on false assumptions.

Having an evaluation plan can also help in the design of the instruction. Take the safety training for an example. If there is a particularly hazardous environment, job aids or monthly safety facts could help reinforce the knowledge and be designed into the instruction plan. Then surprise evaluations can be used to check if this knowledge is being applied.

Another way an evaluation helps is it systemizes and organizes the collection of data. The type of data can be organized to check for satisfaction, learning, application, organizational impact, ROI or other intangibles such as attitude. Then scientific means can be used to gather the required type of data to assess the certain result. From this, we can tighten the measure of success and outcomes.

Are there any ways an evaluation plan can hinder the instruction? It can when the bones of the evaluation plan are still sticking out. I took an online class where the lecturer kept saying, “In this section the student will bla, bla, bla.” As a student it is better to be spoken to personally. Thus, the evaluation plan can be used to create a script, but then the impersonal language needs to be removed to personalize the tome for the audience.

Some other weaknesses are in the collection of data. Sometimes focus group results cannot be generalized. Other times, randomization and control groups are not possible. Also, the collection of data can be time consuming and costly.

However, to ensure effective instruction and programs, it is foolish to not have an evaluation plan, but it should be designed realistically and avoid the pitfalls of over generalizing the data.

Posted in Uncategorized | Leave a comment

Criticisms of Kirkpatrick

Thank you for bringing up the topic of criticisms of Kirkpatrick’s model. I did research into the area and found some things I expected, and some things I didn’t. What I expected was a hornet’s nest as the research is overturning the behaviorism assumptions that 20th century training was based on in light of 21st century research in cognitivist theory, the brain and workplace changes due to technology.

And a hornet’s nest it is, with many falling on the side of Kirkpatrick. It seems as though his theory and company have been very influential. Trainees now expect a smile sheet and a post-test. Managers expect some sort of level 3 and level 4 data. And not many want to change this expected model.

Many realize there are logical holes in Kirkpatrick’s system. Ried Bates gives a nice criticism of the flawed logical assumptions in the model that it assumes causal links and incremental levels. Meanwhile, Stephen Gill points out that much of the data gathered by the model is self-reported. Who doesn’t like to rate 2 days off of work learning as better than working?

In response to these criticisms, there is an updated model of the 4 levels. However, this appears to be more of a window dressing with a flow chart outlining the process of the evaluation. It still contains the same logical flaws.

Another unexpected thing was how Kirkpatrick and Associates has become a cult of personality, a creaky old edifice whose humble, apple pie exterior was unwilling to adapt to modern reality. Then I read Kirkpatrick’s original text and saw stubborn, circular logic.

What I didn’t expect to find was the void of another model.  There are models, but not as clear as Kirkpatrick’s 1, 2, 3, 4 simplicity. One that leverages this is Kaufman, Keller, and Watkins. They add organizational elements and suggest learning ecosystems, but keep the same basic 4 levels.

Other proto-systems would include applying 6 Sigmas to evaluate training, using a modified narrative theory like Brinkerhoff’s Success Case Method, or a system’s view approach such as Stufflebeam’s CIPP evaluation model.

To make a philosophical leap, the basic problem seems to be The Game of Work and The Game of Training, which is based on the Game of Learning. When we break the 18th century idea of schools, training and organizations that we dwell in, we will have to discover the experience, social, mistake driven cognitive reality that is learning, and how to evaluate it, probably not so simplistically and mechanical.

Yes, we are driving around in 54 Dodge’s with the evaluation methods of Kirkpatrick. But it gives us enough seemingly valid justification for it that our dopey leaders, who don’t really understand learning or training, check off on it. So why change? More appropriate, why break the wheel that feeds us?

Bates, R, 2004, A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. http://goo.gl/Up5P3h

Gill, S, 2011, Kirkpatrick’s Four Levels of Training Evaluation: A Critique. http://stephenjgill.typepad.com/performance_improvement_b/2011/05/kirkpatricks-four-levels-of-training-evaluation-a-critique.html

Kaufman, R, Keller, J., Watkins, R. (2007). What works and what doesn’t: Evaluation beyond Kirkpatrick. http://home.gwu.edu/~rwatkins/articles/whatwork.PDF

Kirkpatrick, J, Kirkpatrick, W.K., 2009. The Kirkpatrick Four Levels: A Fresh Look After 50 Years 1959 – 2009.  http://www.kirkpatrickpartners.com/Portals/0/Storage/Kirkpatrick%20Four%20Levels%20white%20paper%20updated%2010%2009.pdf

Posted in Uncategorized | Leave a comment

Digital Games and Simulations

Doesn’t making digital games and simulations sound like fun? Do you think you have learners that could benefit from learning games? I know I do, so I decided to start looking into it.

Why games and simulations? Games are popular! Who do you know that has not played an interesting online game? Studies show they seem to work, and engage the learners more than lecturing to them. The only down part- how to talk the boss into believing that the serious learning doesn’t have to be serious.

What is Gamification? Simply put, using game mechanics for learning. They can be used for either attitude or behavior change. You design them around the learning objectives. They can have enhanced visuals and/or social interaction to engage the learners.

Why do they work? Have you heard of Fogg’s Behavior Model? The idea is that the interaction of motivation, ability and call to action trigger can be balanced into an experience called flow- where the learner is so drawn into the game that they lose the track of time. Here is a model of it:

FoggBehaviorModel What are game mechanics? These are the parts of a game that synchronize together to draw in the learner. Here is a groovy drawing explaining them:

Game elements-mechanics

So there are some of the basics of games and simulations. Hope you share your ideas!

Posted in Uncategorized | Leave a comment

Are online case studies different?

Many of you have sent me questions asking about online case studies. You wonder if they or the quality of learning are different. I hope this post answers those questions.

How they are different. Case studies rely on interaction and discussion. Since online learning uses different communication tools, you can expect the interaction to be different. Relying on Wikis or discussion boards, there can be a lag time between responses. This is not always bad, because this lag can encourage deeper thinking and higher quality discussions. Of course you can choose to require video teleconferencing also. This mimics face-to-face, without as much of the temptation for extra-curricular social activities. So again, this can be a boon.

What you need to stress is that there is a discursive process to negotiate and formulate the answer product of the case study, and let your students know that this is an important factor in their learning.

Should it be synchronous or asynchronous? This really doesn’t matter for learner performance. There are characteristics of each that certain learners will prefer.

Real Time Asynchronous
Immediate feedback

Similar to in class

Requires an agreed on schedule

Less time spent on task

Free to work anytime

Communication is slower and takes more time

Provides scheduling flexibility

Increasing student preference

Should I structure the learning differently? Online case studies usually give more content up front. In a classroom information can be meted out to the student as they work the case. This can be done online if you build a branched scenario from the case. But this is a time intensive effort.

Other different things to structure your case study for online: Give the learner autonomy, give very clear directions and provide them a method for asking timely questions.

Is implementing online case studies different? From my experience, yes- because of tools and timing. The communication tools for online learning are different, and some learners may not be adjusted to typing their thoughts wrather than speaking them. Also, the instructional tools are different. Some case studies will be media intensive, while others are text intensive. One trick for larger case studies is to scaffold or chunk the information across several modules.

Are there best practices? Some of the best practices I have run across are Chickering and Gameson’s 7 Principles of Good Practice.

  1. Encourage contact between the students and faculty
  2. Encourage collaboration
  3. Encourage active learning
  4. Give prompt feedback and consistent communications
  5. Emphasis time on task
  6. Communicate high expectations
  7. Respect diverse talent and ways of learning

I hope this post has cleared up some of your questions about online case studies. Please let me know your thoughts!

Source: Sharon Watson and Jann Marie Sutton, Online: Does the Technology Matter? An Examination of the Effectiveness of Case Method Teaching. Journal of Management Education 2012 36: 802

Posted in Uncategorized | Leave a comment

Using Storyboards For Collaboration

Now it has become almost normal for real time collaboration on documents. Google Docs led the way with being able to share a document space in real time, but now Word also has this function. This has facilitated the ability to cooperate and eliminated the waste of saving several documents. But what is another way teams can collaborate? I suggest storyboards!

Use of storyboards for collaboration. One of the challenges of adult learners is being creative. The book Improving Creativity, Critical Thinking, Communication, and Collaboration has this example storyboard exercise to help learners be able to visualize and sequence ideas (1). Another way is shown by Mark Rice (2).  Whether training future teachers, media people, or even medical personnel to create a scenario, storyboards can help learners to render a scene to understand an interaction, or create an imaginative story! Following is an online tool your learners can use for online storyboard collaboration.

Storyboard That. This is a classic software that is easy to use with its drag and drop interface. Scaling of characters and props is easily done with the drag of a corner. One nice feature is being able to change the rendering of the background scene. Here is an example of one background with 3 different renderings.


But can it be used for learning? Recently working with it to complete a homework, my 11 year old neice said, “Yeah, we use that in school.” But can it be used by adults? Here is an example adult homework about the misunderstanding  between children and adults. But what about for collaboration? Here is the recent collaborative assignment that my partner and I used Storyboard That to design.

Downside to Storyboard That? There are a limited number of elements you can use. Also the end product is not professional looking. But if you want to develop creativity with a simple, easy to use software for a collaborative storyboard, then Storyboard That is a great tool.

Looking for more tools? Here is a list of storytelling tools. And here is a list of storyboard templates for e-learning. Have fun learning together!

Source: 1. Improving Creativity, Critical Thinking, Communication, and Collaboration By Saddleback Educational Publishing, http://goo.gl/Rzlvut

2. Mark Rice, Factors facilitating or impeding older adults’ creative contributions in the collaborative design of a novel DTV-based application. http://goo.gl/z4d09N

Posted in Uncategorized | Leave a comment

Online collaboration

Web 2.0 presents many exciting new tools for learner collaboration. But what are the challenges to overcome, and how to implement and assess online collaborative learning?

Opportunities. It is more exciting and engaging. Research shows that students are more engaged in the topic when they are debating it with peers (1). Also, it works for learning. According to Constructivism, “knowledge is constructed and transformed by learners” (1).

Challenges. Communication, learning curve and organization are the biggest challenges to online collaboration. The biggest one of these is communication, both between the learners’ and instructor and between the learners themselves. Because of the limited bandwidth of electronic communications, misunderstanding can be easy. Also, asynchronous communication, such as on a blog, can be frustrating.

The learning curve of first time online learners can be steep. If there is a mix of people who are experienced with online learning and those not, it can be awkward as the newbie struggles to define their online learner’s voice and/or the technology interface.

Organization is the third challenge. Again, newbies to online learning can struggle to keep all the balls of assignments up in the air.

Implement. There are some best practices to implementing online collaboration, most of these deal with clear communications. First is give the learner clear, well organized expectations. Next, be clear on how to form groups, whether it is self-chosen or by the facilitator. Have a communication plan. If a member of the group is not actively participating, how will you communicate this? Finally, will you be monitoring the work actively or passively? Of course this depends on your role and the assignment.

Assessment. Last, how will you assess the learning? Learners expect you to be clear how they will be assessed. What role should self-assessment or peer assessment play, if any? But most importantly, how will you deal with student accountability? I am sure that many of us have heard stories of the lazy or incapable group member. Thus, a plan for learners’ accountability to both themselves and the other members of the group is important.

Finally… There is much that can be learned in collaboration, but the proper planning of it, taking into account the challenges you may face as an instructor, are important.

Source: (1) Linda Thistlethwaite. Co-creation in classroom, sharing, multi-session creation, long-term co problem solving. Different levels of collaboration, http://literacy.kent.edu/cra/cooperative/coop.html

Posted in Uncategorized | Leave a comment

Leveraging the Power of Google Docs in E-Learning

One of the most powerful tools for e-learning is Google Docs. This is because it provides the ability to collaborate in real-time from different computers.

Here are the advantages for the teacher:

With Google Docs you can create and manage file folders and who has access to them easily. It is much like the folder and file creation of a common computer desktop. Plus the instructor can create the permissions for what files the students can see and access. Here is an assignment tracker for use with Google Docs. A big bonus is that Google Docs is all cloud storage. This means that you don’t have to worry about your or the schools computer crashing and losing the files. And it is accessible with internet access from any computer or mobile device. So no, “I lost my homework” or, “my computer died” excuses.

Here are some cool ways to use Google Docs:

The real power is in collaboration. The learners can have real time feedback; communicate with others in a group; and share knowledge, insights and opinions. And the tool can scale from small projects for a few learners to large, complex projects with many learners. Also, it can fit a range of learners from elementary school to adult education.

Here is an example of how it is used at an elementary school.

Here is an example of how to use it to create a project template for high school, university or adult learners.

Here is a real cool project that utilizes roles for a digital writer’s workshop. Each student takes a different view of the same document in critiquing it.

Last, here is one to boost interactivity using Google Forms.

As a serious educational theorist, it is important to remember the important real life things. Thanks to The Learning Generalist, I now recognize the true power of Google Docs: It can keep track of who to order ice cream for!

Google Docs Useful

Source: The Learning Generalist

How else can Google Docs be leveraged to create fun and meaningful learning? I look forward to hearing some great suggestions from you!

Posted in Uncategorized | Leave a comment