Kirkpatrick's Revised Four Level Evaluation Model

I had an interesting discussion with Clark Quinn on using Kirkpatrick's model in learning processes other than courses. Clark argues that use of Kirkpatrick’s model is only for courses because training is the dominant discussion on their web site. I disagree and wonder if perhaps it is more of a “not invented here” hesitation because advancing concepts to the next level has often been a primary means of moving forward. It might sound good to forget an old model, but if you do not help people relearn, then their old concepts have a nasty habit of reappearing. In addition, training is far more than just courses. So after some heavy reflection I did a rewrite on my Kirkpatrick web page and have listed some of the highlights below.

More than Courses

While some mistakenly assume the four levels are only for training processes, the model can be used for other learning processes. For example, the Human Resource Development (HRD) profession is concerned with not only helping to develop formal learning, such as training, but other forms, such as informal learning, development, and education (Nadler, 1984). Their handbook, edited by one of the founders of HRD, Leonard Nadler, uses Kirkpatrick's four levels as one of their main evaluation models.

Kirkpatrick himself wrote, “These objectives [referring to his article] will be related to in-house classroom programs, one of the most common forms of training. Many of the principles and procedures applies to all kinds of training activities, such as performance review, participation in outside programs, programmed instruction, and the reading of selected books” (Craig, 1996, p294).

Kirkpatrick's levels work across various learning processes because they hit the four primary points in the learning/performance process... but he did get a few things wrong:

1. Motivation, Not Reaction

Reaction is not a good measurement as studies have shown. For example, a study shows a Century 21 trainer with some of the lowest reaction scores was responsible for the highest performance outcomes in post-training (Results) as measured by his graduates' productivity. This is not just an isolated incident—in study after study the evidence shows very little correlation between Reaction evaluations and how well people actually perform when they return to their job (Boehle, 2006).

When a learner goes through a learning process, such as an elearning course, informal learning episode, or using a job performance aid, the learner has to make a decision as to whether he or she will pay attention to it. If the goal or task is judged as important and doable, then the learner is normally motivated to engage in it (Markus, Ruvolo, 1990). However, if the task is presented as low-relevance or there is a low probability of success, then a negative effect is generated and motivation for task engagement is low. Thus it is more about motivation rather than reaction.

2. Performance, Not Behavior

As Gilbert noted, performance has two aspects: behavior being the means and its consequence being the end... and it is the consequence we are mostly concerned with.

3. Flipping it into a Better Model

The four levels are upside down as it places the two most important items last—results, and behavior, which basically imprints the importance of order in most people's head. Thus by flipping it upside down and adding the above two changes we get:

• Result - What impact (outcome or result) will improve our business?
• Performance - What do the employees have to perform in order to create the desired impact?
• Learning - What knowledge, skills, and resources do they need in order to perform? (courses or classrooms are the LAST answer, see Selecting the Instructional Setting)
• Motivation - What do they need to perceive in order to learn and perform? (Do they see a need for the desired performance?)

With a few further adjustments, it becomes both a planning and evaluation tool that can be used as a troubling-shooting heuristic (Chyung, 2008):

The revised model can now be used for planning (left column) and evaluation (right column).

In addition, it aids the troubling-shooting process. For example, if you know the performers learned their skills but do not use them in the work environment, then the two more likely troublesome areas become apparent as they are normally in the cell itself (in this example, the Performance cell) or the cell to the left of it:

• There is a process in the environment that constrains the performers from using their new skills, or
• the initial premise that the new skills would bring about the desired change is wrong.

The diagram below shows how the evaluation processes fit together:

Learning and Work Environment

As the diagram shows, the Results evaluation is of the most interest to the business leaders, while the other three evaluations (performance, learning, and motivation) are essential to the learning designers for planning, evaluating, and trouble-shooting various learning processes; of course the Results evaluation is also important to them as it gives them a goal for improving the business. For more information see Formative and Summative Evaluations.

I go into more detail on my web page on Kirkpatrick is you would like more information or full references.

What are your thoughts?

Visualization (Sensemaking) in Rapid Agile Learning Design

Common definitions of visualization usually read something like, “to form a mental image,” thus we often think of visualization as being a simple solo technique, such as picturing “a dog eating a bone” or “a person doing the right thing.” However in an organization context, visualization is much more complex in that while it involves an image of the working environment, it is also a complex process that is very social in nature.

The Visualization Framework

Visualization is often used interchangeably with sensemaking—making sense of the world we live and operate in, and then acting within that framework of understanding to achieve desired goals. Thus visualization is not just a shared (social) image with intent, it also implies ACTION. This framework can be used for building agile or rapid learning designs, fixing performance problems, implementing informal learning solutions, etc.

Visualization Framework

Visualization Framework (opens larger image in a new window)

The start of a visualization process is often sparked by a cue from the environment, such as an increase in customer complaints; or a team charged with improving a process. The steps within the visualization or sensemaking framework include (Leedom, McElroy, Shadrick, Lickteig, Pokorny, Haynes, Bell, 2007):

1. Triggering cues (information that acts as a signal) from the environment are perceived by the people in a Community of Interest (CoI). These cues may be picked up by one or more members of the CoI. A couple of examples of triggering cues might be an increase in the number of customer complaints or an unexpected drop in production.

2.Triggering cues create a situational anomaly—facts that do not fit into the framework of familiar mental models. Detection of these anomalies violate the expectancies of the members of the CoI and creates a need for change (improvement).

Note: A mental model is a structure or frame that is built from past experience and becomes part of an individual’s store of tacit knowledge. It is comprised of feature slots that can be instantiated by information describing a current situation (such as triggering cues). Its functional purpose allows a person to assess the situation, take a course of action, follow causal pathways, and recognize constraints in order to achieve a set of goals for actively confronting the situation. Fragmentary mental models can often be linked together to form a just-in-time explanation of a situation. Examples of a mental model include a chess player reacting to a move on the chessboard, a doctor diagnosing a medical condition, or an instructional designer solving a performance problem.

3. Specific data from the information environments trigger the mental activation of familiar mental models. The members of the CoI analyze and discuss the anomalies until they discover a purposeful structure or pattern for interpreting the new information. This transforms the problem space into various solutions. This process of “pattern matching” starts the basis for constructing new or revised mental models. Since patterns differ among the members, they collaborate by telling stories, metaphors, etc. to build common understanding.

4. Activation of a specific mental model is typically triggered by matching salient facts to one or two key features that uniquely anchor a new model that the CoI can agree upon. Tacit knowledge or intuition is often used to build mental models and the degree of tacit knowledge will vary among the members, thus they use a “negotiation process” to ensure all needs are met (or at least prioritize them according to available resources).

5. An action plan is used to instill the selected mental model into the work space in order to transform it to the desired state (during the visualization process intent must always be associated with action, otherwise it is just wistful thinking). The action plan includes the final development of any needed content, material, or products. Once all the pieces are put together, the action plan is implemented.

6. New information from the transformation process is perceived by the CoI, which in turn processes it to determine if the patterns match their desired mental model.

7. If the new information does not match the CoI's newly constructed mental model (situational anomalies are again perceived and they may or may not differ from the original ones), then the visualization process begins anew.

Probing and Shaping

While the visualization process does use passive information that derives from experience and expertise, it also involves the proactive use of shaping actions to reduce risk and uncertainty and probing actions to discover system effect opportunities that can then be exploited.

Probing develops greater understanding by experimentally testing the operational environment, such as asking questions, Cognitive Task Analysis, or immersing oneself in the troubled environment to discover new information. These probing actions help to illuminate key structures and linkages within the environment.

Shaping is taking an incentive action to discover new information in order to determine if it aids in transforming the troubled environment to meet the new mental model. Prototyping may be used as a shaping tool—an iterative process of implementing successive small-scale tests in order to permit continual design refinements. There are normally two types of prototypes:

• Design Iteration (interpretive) — the iteration is performed to test a learning method, function, feature, etc. of the action plan to determine if it is valid.
• Release Iteration (statistical) — the iteration is released as a product to the business unit or customer. Although it may not be fully completed or functional, the designers believe that it is good enough to be of use.

Probing actions serve to illuminate additional elements and linkages within the visualization space that can then be subsequently exploited for operational advantage.

Visualization is Dynamic, Not Static

The visualization or sensemaking framework in not linear, but rather a dynamic process that may flow in any direction, for example:

The Dynamics of Visualization

Dynamics of the Visualization Framework (opens larger image in a new window)

A Community of Interest holds a vested interests when faced with a troubling situation, thus they need a dynamic model that aids them in fulfilling their mission within complex environments. The military has a term called “center of gravity,” which is defined as the source of power that provides moral or physical strength, freedom of action, or the will to act. It is seen as the source of strength of the organization. The ability to act upon and transform an under-performing environment through the use of visualization or sensemaking is an essential attribute in an rapidly moving environment in that it helps to ensure the center of gravity stays balanced.

Reference

Leedom, D. K., McElroy, W., Shadrick, S. B., Lickteig, C., Pokorny, R. A., Haynes, J. A., Bell, J. (2007). Cognitive Task Analysis of the Battalion Level Visualization Process. Arlington, VA: United States Army Research Institute for the Behavioral and Social Sciences. Technical Report 1213. Retrieved on January 5, 2012 from http://www.hqda.army.mil/ari/pdf/TR1213.pdf

Learning Styles are for the individual, not group

NOTE: I left this comment in eLearn Magazine's, Why Is the Research on Learning Styles Still Being Dismissed by Some Learning Leaders and Practitioners by Guy Wallace. Since it wiped out most of my formatting, such as comments and quotation marks, I am posting it here for better readability.

Perhaps one of the best papers on learning styles is Coffield, Moseley, Hall, and Ecclestone's, Learning styles and pedagogy in post-16 learning: A systematic and critical review (PDF). While the paper does dismiss some types of learning styles and the importance that the recognized learning styles actually have when it comes to learning, it does leave a lot of questions opened.

One of the most profound statements in the paper, at least to me, is (p68):

“just varying delivery style may not be enough and... the unit of analysis must be the individual rather than the group.”

That is, when you analyze a group, the findings often suggest that learning styles are relative unimportant, however, when you look at an individual, then the learning style often distinguishes itself as a key component of being able to learn or not. Thus those who actually deliver the learning process, such as teachers, instructors, or trainers and are responsible for helping others to learn see these styles and must adjust for them, while those who design for groups or study them see the learning styles as relative unimportant.

In the next paragraph, the paper continues with this statement:

“For each research study supporting the principle of matching instructional style and learning style, there is a study rejecting the matching hypothesis’ (2002, 411). Indeed, they found eight studies supporting and eight studies rejecting the 'matching' hypothesis, which is based on the assumption that learning styles, if not a fixed characteristic of the person, are at least relatively stable over time. Kolb's views at least are clear: rather than confining learners to their preferred style, he advocates stretching their learning capabilities in other learning modes.”

While many find this as a reason to dismiss learning styles, I find it quite intriguing in that why do learning styles play a key component is some situations or environments, but not others? I think part of the answer is within this finding—a study that was conducted in the U. S. and Israel, found that when students' learning styles matched the teaching method they performed both more effectively and efficiently. But the authors of the paper seem too readily to dismiss it as the end the paragraph with this statement—“But even this conclusion needed to be qualified as it applied only to higher-order cognitive outcomes and not to basic knowledge.” (p67)

It seems logical that higher-order cognitive outcomes need more individual support (in this case matching the learning style the the correct learning strategy) than basic knowledge. Thus in some situations learning styles are important, while in others they are not.

Finally, in the paper's conclusion the authors note (P132-133) that:

“Despite reservations about their model and questionnaire (see Section 6.2), we recognise that Honey and Mumford have been prolific in showing how individuals can be helped to play to their strengths or to develop as all-round learners (or both) by means, for example, of keeping a learning log or of devising personal development plans; they also show how managers can help their staff to learn more effectively.”

Thus the main take-away that I get from the paper if that if you are an instructor, manager, etc. who has to help the individual learners, then learning styles make sense. On the other hand, if you are an instructional designer or someone who directs her or his efforts at the group, then learning styles are probably not that important. Note that I am both a trainer and a designer so perhaps this is why my take-away makes sense to me.

Lingering Doubts About the 70:20:10 Model

In a couple of recent posts both Ben Betts and Clive Shepherd casts their doubts about the usefulness of 70-20-10 model and wonder if it's confusing the issue. You can read their posts at The Ubiquity of Informal Learning, and Beware who's selling informal learning.

I tend to agree with them, but before I begin I want to add that if you think I'm anti-informal learning, then please note that I wrote a post defending informal learning and it was Tweeted quite heavily. In addition, I've seen in the comments of these posts and others that if you challenge the idea about the usefulness of the 70-20-10 model then either you don't want to understand it, you clearly don't get it, or you see it as a threat to your job. If this is what you really think then you may talk-the-talk of informal and social learning learning but you walk-the-walk of a lecturer—“it's my way or the highway.” I have no patience with these attitudes because they are simply attacking people rather than their ideas.

While some proponents of the model insist it is non-prescriptive, both Ben Betts and Clive Shepherd saw the model as being “prescriptive.” I saw it as being prescriptive. Jay Cross saw it the same way as he wrote in one of his posts, A model of workplace learning,“The 70-20-10 model is more prescriptive. It builds upon how people internalize and apply what they learn based on how they acquire the knowledge.”

Even the Center for Creative Leadership, where the model was developed, write that the 70-20-10 model is indeed prescriptive:

“A research-based, time-tested guideline for developing managers says that you need to have three types of experience, using a 70-20-10 ratio: challenging assignments (70 percent), developmental relationships (20 percent) and coursework and training (10 percent).”

The 70-20-10 model is a prescriptive remedy for developing managers to senior and executive positions. Parts or perhaps all of the model may be useful for developing other professionals. However, by no means is it a useful model for the daily learning and work flows that takes place within organizations because it is being applied in an entirely different context that what it was designed to do. When people see numbers applied to a model they normally assume a couple of things: 1) that it is fact based, and/or 2) this is the way it is supposed to be.

As Will Thalheimer noted in one of his posts, adding numbers to make a model look more authentic makes it both bogus and dangerous (see People remember 10%, 20%...Oh Really?). I can attest to that because in some of my posts in the past I wrote that the formal to informal ratio was 30/70. People immediately commented and insisted it was 20/80 or 10/90. They seemed determined to lock the numbers in to an exact ratio—NO EXCEPTIONS! Even the model is begriming to look more like real ratios that must be adhered to because it is now being written as 70:20:10. Where will it end?

For more on the ratios see, 70-20-10: Is it a Viable Learning Model?

Yes, you can manage informal learning

Jane Hart recently posted a thought-proving article on her blog in which she argues that you “can't manage informal learning, you can only manage the social media tools.” In her post she goes to great depth to define some of the various types of learning, such as formal, non-formal, and informal, however, I think the same needs to be done for “manage” in order to get a more accurate picture, otherwise we get mental images of Dilbert's pointy-haired boss when someone speaks or writes about management.

People often equate the term “management” with “control,” that is, when you manage something, you are trying to take direct control of it. However, management and control are actually two of four distinct processes for guiding an organization. The other two are leadership and command. While these are separate processes, they need to be blended together to deal with our rapidly changing world. Note that while I define the terms based upon my military experience and training, civilian organizations often use them because the military has the resources to study and research these concepts (and their studies are often done on civilian organizations which makes them valuable to the outside world).

Command and Control

Command is the imparting of a vision to the organization. It does this by formulating a well-thought out vision and then clearly communicating it. It emphasizes success and reward. That is, the organization has to be successful to survive and in turn reward its members (both intrinsically and extrinsically).

An example in this case would be visioning a process that helps to increase informal learning and make it more effective. A bad vision would be implementing a social media tool, such as a wiki or Twitter. This is because they are tools or technologies that are means rather than an end-goal.

Visions do not have to come from the top, but rather anywhere in the organization. Informal leaders are often good sources of visions, however if the vision requires resources, then they normally need the support of a formal leader.

In contrast, control is the process used to establish and provide structure in order to deal with uncertainties. Visions normally produce change, which in turn produces tension.

For example, “is the tool we provided to increase the effectiveness of informal learning really working?” Thus it tries to measure and evaluate. Inherent in evaluation is efficiency—it tries to make the goal more efficient. This can be good because it can save money and often improve a tool or process. The danger of this is if the command process is weak and the control process is strong then it can make efficiency the end-goal. That is, it replaces effectiveness with efficiency.

A good example of this is our present economy that caused many organizations to perform massive layoffs. Now the same organizations are complaining that they can't find qualified workers. Efficiency over road effectiveness—they failed to realize that they would need a trained workforce in the future.

Leadership and Management

Management's primary focus is on the conceptual side of the business, such as planning, organizing, and budgeting. It does the leg work to make visions reality. Thus it helps to acquire, integrate, and allocate resources to accomplish goals and task. This is why you need to manage non-formal learning and not just the tool itself. The goal is to increase informal learning and make it more effective, not to put into place a media tool. If the tool because the goal, then the wrong polices could be put into place that decrease its value as an informal learning tool.

Secondly, if the focus is only on the tool, then other options are omitted, such as tearing down cubicles and creating spaces where people can meet.

In contrast, leadership deals with the interpersonal relations such as being a teacher and coach, instilling organizational spirit to win, and serving the organization and workers.

Thus all four processes have their place. When you manage informal learning, you are not trying to control it, but rather planning how you will put the vision in place, budgeting for the required resources, and then organizing the teams so they can make it a reality.

In the August 2010 (p.10) edition of Chief Learning Officer magazine, Michael Echols notes a survey that the number one priority of 96 percent of the CEOs they surveyed want proof that learning programs are driving their top five business measures, but only 8 percent are getting it. Thus the learning and development leaders are going to start feeling the heat to get some type of evaluation process into place. If informal learning is going to be one of the primary objectives, we are going to have to get real about actually trying to measure it. The excuse that the learners control it so it can't be done is not going to cut it for long.

Mapping the Performance

Part of the analysis phase is capturing the skills required for performance. While there are several methods for capturing these performance, ISD normally only lists Behavioral Task Analysis. However, many task are largely overt and nonprocedural in nature, thus they require a Cognitive Task Analysis. The solution of course is to simply plug the desired method or tool into the ISD or ADDIE model as it is quite dynamic rather than the stale linear model that some believe.

Four analysis tools will be discussed in this post that may be plugged into ISD:

• Behavioral Task Analysis
• Information Processing Analysis
• GOMS Analysis
• Critical Decision Method

Considerations for Analysis Tool Selection

Selecting the correct analysis tool is dependent on the type of actions the worker must perform. This performance is normally composed of two types of actions:

• Overt - behavioral and observable
• Covert - mental and not observable

While some some tasks are only composed of one or the other, more complex tasks may be composed of both actions.

In addition, the selection of the analysis tool is also dependent upon the task steps:

• Procedural - the steps are performed in order and are normally largely overt actions
• Rule Based - the steps do not have to be performed in a temporal order and are normally largely covert (cognitive) actions

Procedural Analysis Methods

These methods are used when there is a temporal order of involved steps, thus there is a set procedure for performing the task. Two analysis tools that fall under Procedural Analysis are Behavioral Task Analysis and Information Processing Analysis.

Behavioral Task Analysis

Behavioral Task Analysis is used to capture overt actions by observing and recording an Exemplary Practitioner perform the task. Questions may also asked to ensure the analyst has fully captured the performance. This is perhaps the easiest method. The output is a list of steps that may also have diagrams or pictures of the desired performance or behavior. A short example is:

1. Turn on computer and start spreadsheet application.
2. Load projected sales report spreadsheet template (prosale.exl).
3. Enter projected sales figures into designated spreadsheet cells.
4. Run spreadsheet macros.
5. Save file under new name, “pro***.exl”, with *** being the next sequential number For example, pro135.exl. Note: Do NOT overwrite template.
6. Forward to Planning Manager by email.
7. Exit application.

Depending upon the learners' prior knowledge and the complexity of the task, the list might also contain substeps as shown in this example:

1. Turn on computer
2. Start spreadsheet application
• 2.1 Click the Start icon
• 2.2 Scroll through the menu list and select the Excel application
3. Load projected sales report spreadsheet template (prosale.exl).
   • 3.1 Click on the file menu
   • 3.2 Click on the Open option
   • Etc., Etc.

Information Processing Analysis

This tool is used when there are both overt steps that require a set order and covert steps that require decision making of a yes or no nature. The overt actions are captured by observing and recording an expert performer, while the covert actions are captured by having the expert performer talk about their actions (thinking aloud). Capturing the decision making process makes it more difficult than the Behavioral Task Analysis, but it is perhaps the most common method as the majority of tasks require decisions. The output is commonly a flow chart composed of three elements that outline the task steps:

• Boxes - overt or covert actions
• Diamonds - decisions
• Arrows - order of steps

A short example of a flowchart showing a decision a forklift operator must make when moving goods from the receiving dock to a storage area may look like this:

Rule Based Analysis Methods

These methods are used when there is NO temporal order of involved steps, thus there is not a set procedure for performing the task. In addition, most of the task steps are normally of a overt nature. Two tools that fall under Rule Based Analysis are GOMS Analysis and the Critical Decision Method.

GOMS Analysis

The GOMS tool analyzes a task by examining four elements of it:

• Goals represents the intention to perform a task its underlying structures, such as subtasks, cognitive operations, and physical operations. For example, an Instructional Designer may have a task of selecting activities that will give a new sales person the skills to sell a product. That task includes subtasks of analyzing the skills needed, analyzing activities, etc. The ID will use cognitive operations, such as selecting an activity that actually teaches the skill and determining if the chosen media can effectively deliver the activity.
• Operations represent the physical actions, such as binding a learner's manual or pressing a button. It also includes mental operations, such as retrieving from memory or setting a goal.
• Methods represent sequences of operations that accomplish goals or objectives. It includes high-level goals that breaks a tasks into subtasks and low-level methods that are the actions that actually perform the subtasks.
• Selections Rules represent the context for selecting a particular method. That is, there may be several ways to accomplish a goal or task, but one may be chosen because it should perform this particular task the best (heuristics).

Capturing the performance of a covert task can be done in many ways and normally several are used to capture the task. Some of the more common ones are interviews, job shadowing, having an expert performer think-aloud, and storytelling.

The GOMS analysis can normally best be represented by a concept or mind map. The Goals will normally be placed on the map first, which is then followed by placing the Methods and their Selection Rules on the map. The details (Operations) are then listed. Shown below is a partial mind map of selecting an analysis tool:

Depending upon the size and the scope of the task, you might have to link or reference other documents that go into more specifics about the Operations, Methods, and Selections rules. However, be careful as one of the common mistakes in GOMS analysis is getting too specific, which results in long and detailed procedural descriptions that are difficult to follow.

Critical Decision Method (CDM)

This can be thought of as a Case Study, however, it also includes a visual reference or map. Just as a case study uses an actual incident to tell a story, CDM is also performed by having an expert tell a story about a particular task they performed in the past. For example, an Instructional Designer might tell how he developed a Just-in-Time program for training sales persons to sell a new product or a fire fighter might tell about the actions she took in fighting a gas station fire. Note that when the person tells the story, the interviewer might have to probe to gather some details. While the output might include a case study, it should normally include a map similar to this:

The CDM process goes like this:

• Sweep 1 - Identify a incident. It should come from a decision maker who was involved rather than a witness.
• Sweep 2 - The expert tells his or her story. Identify the key decision points and when they were made.
• Sweep 3 - These is where you “deepen” the interview by asking for analogs, mental models, other options, experience or training that was helpful, etc.
• Sweep 4 - Finally ask “What If” questions, such as “If the situation had been different what would have happened?”, or “What if a novice had been in charge?”

Key Points

The finale outputs, such as a list of task steps, flowchart, mind map, or CDM chart prove not only invaluable to the the design team, but may also make excellent performance or learning aids. Just as it is as important to use a correct analysis tool, it is also just as import to represent the findings in a manner that others can understand.

The final output might not be just one chart, but rather a combination of them as shown in the picture at the top of this post. Also the chart does not have to be combined into one page if it gets too cluttered, but may rather be composed of a collection of documents that are linked together electronically or referenced if they are printed. In addition, other types of charts and visualizations representations may also be used.

More detailed information can be found in two books, van Merriënboer's, Training Complex Cognitive Skills: A Four-Component Instructional Design Model for Technical Training (1997) and Crandall, Klein, & Hoffman's, Working Minds: A Practitioner's Guide to Cognitive Task Analysis (2006). Merriënboer discusses Behavioral Task Analysis, Information Processing Analysis, and GOMS Analysis (plus other topics related to complex tasks), while Crandall, Klein, & Hoffman discusses Concept Mapping and Critical Decision Method (plus other topics related to Cognitive Task Analysis).

Both Merriënboer and Crandall, Klein, & Hoffman's books are excellent references and provide different information about Cognitive Task Analysis, however, Merriënboer's is now out of print, thus the used books may be a bit on the pricey side.

ADDIE Does More Than Classrooms

One of the misconceptions of ISD is that it was created to only build classroom training environments.Yet, one of the old Army manuals (1984) that is used for training ISD shows the above options for training. It also notes that options 1, 2, or 3 should be used in lieu of classroom training if it can adequately perform the job. Thus, classroom training should normally be the last option if there is more than one viable option. This is because classroom training is normally one of the more expensive options.

The five options are shown below with a few notes about them:

1. Job Performance Aid (JPA) - this would include today's EPSS (Electronic Performance Support System)
2. Self-Teaching Exportable Package - the elearning we know today would fall under this category
3. Formal On-the-Job-Training (OJT)
4. Installation Support School (on or near the employees' workplace) - this would be formal classroom training even though the training may be conducted outside in the field
5. Resident Instruction (away from employees' workplace where travel and living expense would have to be considered) -this would also be formal classroom training.

Just as training designed for the classroom, the other options also need to follow the five phases of ISD to ensure they do what the are supposed to do. For example, even a simple JPA requires:

• Analyzing the various settings and media to determine if it is the most approximate method.
• Designing it so it performs as intended.
• Developing it into a real product.
• Delivering (implementing) it to the workers who need it.
• Evaluating it to ensure it does the job it was intended to do. This also shows the business units that you care about the solutions you deliver (if it ain't worth following up on then it probably ain't worth doing) and you might learn something. Note the evaluation may be as simple as checking with a couple of managers and some of the employees to ensure it is doing what it is supposed to do.

The manual also gives some guidelines for selecting the correct training setting:

• Job Performance Aid
  • close supervision not required
  • task follows a set procedure
  • JPA can be followed while performing the task
  • do not use if:
    • consequence of inadequate performance is high
    • employee lacks prerequisite skill
    • task requires high psychomotor skills
• Self-Teaching Exportable Package
  • close supervision not required
  • task can be self-learned by individual or groups
  • material required for training can be adequate packaged
  • do not use if:
    • task failure would result in injury or damage
    • special facilities or equipment required
• Formal On-the-Job-Training
  • close supervision is required
  • task can be self-learned by individual or groups in the workplace
  • task decay rate is very high
  • do not use if:
    • sufficient equipment is not available for learners to practice on
    • workplace cannot absorb the learners adequately
    • training would be disruptive to normal operations
• Classroom
  • large group must be taught the same thing
  • task difficulty requires a high state of training (task is difficult and requires time to acquire skills)
  • learner interaction is required (such as team training)
  • material required for training cannot economically be placed in the field
  • essential the employee be able to perform upon job entry (high consequence if employees are inadequate performers)
    • do not use if:
    • task may be adequately trained elsewhere

In addition, think blended learning. When I was first trained in ISD we called it BoB (Best of Breed). Blended learning solutions are normally more efficient and effective when designed correctly as they inherit the best of each setting. And do not think of blended as just Brick and Click, but rather any combination of the above, plus more informal options, such as mentoring and social learning media.

Related Posts:

ADDIE Backwards Planning Model

ADDIE and the 5 Rules of Zen

ID is not ISD

One of the trends in the learning industry is proclaiming that a new Instructional Design (ID) model, such as rapid development prototyping, needs to replace Instructional System Design (ISD) because the new model provides more benefits, such as it's newer, dynamic, and faster. Yet ID models differ from ISD models, thus its sort of like saying that a new boat model is going to replace the automobile—yes they are both transportation devices but they do differ in their uses!

ID (Instruction Design) models differ from ISD models in that ISD models have a broad scope and typically divide the instruction design process into five phases (van Merriënboer, 1997):

• Analysis
• Design (sometimes combined with Development)
• Development
• Implementation or Delivery
• Evaluation

Since ISD models cover a broad spectrum they normally do not go into much detail in the design phase. This is where ID models excel. Since they are less broad in nature and mostly focus on design, they normally go into much more detail for the design phase.

Two popular ISD models are ADDIE and The Dick and Carey Model. ISD can also be extended by using Frog Design's model to solve wicked or complex problems as it aligns with ADDIE:

Some popular ID models include Rapid Instructional Design (RID), Gagne's Nine Steps of Instruction, John Keller's ARCS model, Merrill's Component Display Theory, and van Merriënboer's 4C/ID Model.

ISD can be thought more of as a project management tool while ID models are specialized tools used to enhance the learning process.

Omitting ISD and relying strictly on an ID model often omits critical parts of the design process, such as analysis and evaluation. Thus, unless you design for certain groups in an organization or industry in which you know your learners, analysis is important to determine the skill level that the learning program is aimed at. In addition, managers will often identify any performance problem as a training problem, thus the designer needs to ensure it is indeed a training problem rather than a bad process or motivation problem.

Evaluation is not only important to determine if the program is meeting the needs of the organization, but also as a learning tool for the designers themselves.

The best way to use ID models is to plug them into the ISD model as they are needed. For example:

This method allows you to gain the benefit of the ID model that will best suit your needs for enhancing your learning program, while ensuring that your learning program will do what it is supposed to do.

Reference

van Merriënboer, J. J. G. (1997). Training Complex Cognitive Skills: A Four-Component Instructional Design Model for Technical Training. Englewood Cliffs, New Jersey: Educational Technology Publications.

A Look Behind Robert Gagnè's Nine Steps of Instruction

In her post, Questioning Gagnè and Bloom’s Relevance, Christy Tucker describes how we often get caught up in theories without really looking at whether the research supports those theories. In this post, I would like to point out some of the research and newer findings.

While some think the Nine Steps are iron clad rules, it has been noted at least since 1977 (Good, Brophy, p.200) that the nine steps are “general considerations to be taken into account when designing instruction. Although some steps might need to be rearranged (or might be unnecessary) for certain types of lessons, the general set of considerations provide a good checklist of key design steps.”

1. Gain attention

In the military we called this an interest device—a story or some other vehicle for capturing the learners' attention and helping them to see the importance of learning the tasks at hand. For example, when I was training loading and and unloading trailers with a forklift, I would search the OSHA reports for the latest incidence report on a forklift operator who decapitated themself by sticking their head out of the protective structure of the forklift cage in order to get a better view when entering the trailer and then getting it caught between the bars supporting the forklifts protective top and the side of the trailer (it happens more often than we care to think about). This would become the basis for a story on why they needed to pay attention as the forklift may be small, but it weighs several tons and can easily slice off a limb or another body part if not treated with proper respect.

Wick, Pollock, Jefferson, and Flanagan (2006) describe how research supports extending the interest device into the workplace in order to increase performance when the learners apply they new learnings to the job. This is accomplished by having the learners and their managers discuss what they need to learn and be able to perform when they finish the training. This preclass activity ends in a mutual contract between the learners and managers on what is expected to be achieved from the learning activities (this is also closely related to the next step).

2. Tell learners the learning objective

Marzano (1998, p.94) reported an effect size of 0.97 (which indicates that achievement can be raised by 34 percentile points) when goal specification is used. When students have some control over the learning outcomes, there is an effect size of 1.21 (39 percentile points). This is the beauty of using Wick, Pollock, Jefferson, and Flanagan's mutual contract.

Of course the problem that some trainers and instructional designers run into is telling the learners the Learning Objectives word for word, rather than breaking it down into a less formal statement.

3. Stimulate recall

This is building on prior learning and forms the basis of scaffolding by 1) building on what the learners know, 2) adding more details, hints, information, concepts, feedback, etc. 3) and then allowing the learners to perform on their own. Allan Collins John Seely Brown, and Ann Holum (1991) note that scaffolding is the support the master gives apprentices in carrying out a task. This can range from doing almost the entire task for them to giving occasional hints as to what to do next. Fading is the notion of slowly removing the support, giving the apprentice more and more responsibility.

Part of stimulating recall is having the learners take notes and drawing mind maps. Learning is enhanced by encouraging the use of graphic representations when taking notes (mind or concept maps). While normal note taking has an overall effect size of .99, indicating a percentile gain of 34 points, graphic representations produced a percentile gain in achievement of 39 points (Marzano, 1998). One of the most effective of these techniques is semantic mapping (Toms-Bronosky, 1980) with an effect size of 1.48 (n=1), indicating a percentile gain of 43 points. With this technique, the learner represents the key ideas in a lesson as nodes (circles) with spokes depicting key details emanating from the node.

4. Present the stimulus, content

Implement (nuff said)

5. Provide guidance, relevance, and organization

Kind of redundant as it relates to the other steps.

6. Elicit the learning by demonstrating it (modeling and observational learning)

Albert Bandura noted that observation learning may or may not involve imitation. For example if you see someone driving in front of you hit a pothole, and then you swerve to miss it—you learned from observational learning, not imitation (if you learned from imitation then you would also hit the pothole). What you learned was the information you processed cognitively and then acted upon. Observational learning is much more complex than simple imitation. Bandura's theory is often referred to as social learning theory as it emphasizes the role of vicarious experience (observation) of people impacting people (models). Modeling has several affects on learners:

• Acquisition - New responses are learned by observing the model.
• Inhibition - A response that otherwise may be made is changed when the observer sees a model being punished.
• Disinhibition - A reduction in fear by observing a model's behavior go unpunished in a feared activity.
• Facilitation - A model elicits from an observer a response that has already been learned.
• Creativity - Observing several models performing and then adapting a combination of characteristics or styles.

7. Provide feedback on performance

As Christy's post noted, performance and feedback are good.

8. Assess performance, give feedback and reinforcement

Related to above.

9. Enhance retention and transfer to other contexts

We often think of transfer of learning as just being able to apply the new skills and knowledge to the job, but it actually goes beyond that. Transfer of learning is a phenomenon of learning more quickly and developing a deeper understanding of the task if we bring some knowledge or skills from previous learning. Therefore, to produce positive transfer of learning, we need to practice under a variety of conditions. For more information, see Transfer of Learning.

References

Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: Making thinking visible. American Educator, 6-46.

Good, T. & Brophy, J. (1990). Educational Psychology: A realistic approach. New York: Holt, Rinehart, & Winston.

Marzano, Robert J. (1998). A Theory-Based Meta-Analysis of Research on Instruction. Mid-continent Aurora, Colorado: Regional Educational Laboratory. Retrieved May 2, 2000 from http://www.mcrel.org/products/learning/meta.pdf

Wick, C., Pollock, R., Jefferson, A., Flanagan, R. (2006). Six Disciplines of Breakthrough Learning: How to Turn Training and Development Into Business Results. San Francisco: Pfeiffer

Andragogy vs. Pedagogy

In his post, Learning is learning, Steve Wheeler asks, “So does the concept of Andragogy add any value to our understanding of learning? For me, the answer is no.”

I would have to disagree because the concept of andragogy has actually added great value to our understanding of learning.

Pedagogy is derived from the Greek words paid meaning “child” and agogus meaning “leader of.” In this pedagogy classroom, the teachers are responsible for all decisions about learning in that they decided what is to be learned, how it is to be learned, when it should be learned, and if it has been learned. Which meant the learners were pretty much in the roles of passive, dependent recipients of the teachers' transmissions. When our public schools were first established, they were based on this pedagogical model.

When adult education was later established, this was the only model at the time, so our profession was also based on it. Which of course lead to a high drop out rate, low motivation, and poor performance. In 1926, Eduard C. Lindereman's book, The Meaning of Adult Education, captures the essence of adult learning:

In this process the teacher finds a new function. He is no longer the oracle who speaks from the platform of authority, but rather the guide, the pointer-out who also participates in learning in proportion to the vitality and relevance of his facts and experiences. In short, my conception of adult education is this: a cooperative venture in nonauthoritarian, informal learning, the chief purpose of which is to discover the meaning of experience; a quest of the mind which digs down to the roots of the preconceptions which formulate our conduct; a technique of learning for adults that makes education coterminous with life and hence elevates living itself to the level of adventurous experiment. - quoted in Nadler, 1984, p.6.4

In the 1950s, European educators started using the term “andragogy,” from the Greek word anere for “adult,” and agogus, “the art and science of helping students to learn.” They wanted to be able to discuss the growing body of knowledge about adult learners in parallel with pedagogy.

Andragogy, is often criticized because as we now know, it also applies to younger learners; however the people behind the theories at the time were trainers of adults rather than educators in the school system, thus they applied their theories to the section of the population that they best knew about. Because of their work, they pioneered the way for the world of pedagogy to also advance itself from being almost entirely passive-based to a more experience-based process of learning.

So yes, Knowles's concept of andragogy is that he intended for it to be different to pedagogy, because pedagogy at the time was extremely passive-based. Just because pedagogy is finally catching up to andragogy is not a strong enough reason to drop the concept from our terminology. I believe we should be embracing the term because of its rich history and pioneering the way of our present concept of learning.

Reference

Nadler, Leonard (1984). The Handbook of Human Resource Development. New York: John Wiley & Sons