If it can be measured, it can be valued

Training is often viewed as a cost by companies and therefore is on the chopping block when costs need to be cut. Much like the arts in K12 education, training is seen as a "nice to have," but not a core function of the organization.

Why is training viewed as a cost and not a strategic investment? Because learning professionals don't measure the impact of training on the performance of the organization.

Think about the last time you delivered a training program in an organization. Or the last time you participated in training, if you aren't an L&D professional. How was the success of the training measured? 

How many completed the training?

A passing score on an assessment? 

A subjective "smile sheet" ranking where learners reported how valuable they thought the training was?

Do these training programs measure improvements in employee performance? Decreased time to market/increased productivity? Customer satisfaction ratings? Increased revenue? Reduction in costs? Increased employee retention?

No? WHY NOT?

If training is a strategic investment, then it is absolutely critical that we show the value of the investment. Attendance is not value; increased sales is. Assessment scores are meaningless if people don't perform their jobs better. 

If our role in an organization is to improve performance, then shouldn't we be measuring performance? 

Only when we start measuring the right things will training be valued. 

Learning through "infuriating feedback"

One of the things that I'm most passionate about doing well when designing learning is providing feedback. In immersive design, practice without feedback is useless, and often can reinforce the wrong behaviors. For example, if you're trying to perfect your golf swing, heading to the driving range and hitting a bucket of balls without the guidance of a golf pro to help critique your form may lead you to practicing a whole bucket full of bad swings. 

Yet very rarely do any of us perform a task in a vacuum; we're constantly bombarded with feedback, both subtle and obvious, that reinforces what we're doing well and discourages us from continuing less than stellar behaviors. We love hearing the good stuff, getting that positive reinforcement. In game design, quick victories are by design to make players feel good, feel confident, hoping to hook them in to continue playing even as game play becomes more difficult and complex, and victories and positive reinforcement are harder to come by. In "real life," the same dynamic exists. I love to hear the feedback of people after I speak, to hear how something I said struck a chord or helped them make a connection or see something in a different light. Even better, I love when people compliment my kiddos, as that provides me with some positive evidence from non-biased sources that I'm doing ok in the toughest job that I have.

Negative feedback, critique and even punishment surround us every day too. I am always amazed as a mom about how often I have to say, "chew with your mouth closed," or "put some clothes on," or "put your dirty laundry in your hamper, not just on the ground next to it." More subtle things, like a look from my boss or my husband, are enough to make me pause and consider what I just did to elicit that reaction. In game design, negative feedback design creates for interesting play dynamics; losing points, finding yourself in a death spiral that you just have to wait out, or an abrupt game over when you make a bad decision are all ways to provide critique to your play performance and prompt you to try again and do better.

This continuous flow of positive and negative feedback help us learn and shape our future behavior. Other factors contribute to our decision-making, but ultimately, it is the balance of potential risk and reward that are in constant competition to determine the decisions that we make, and we depend on this river of feedback to help us determine if we're on the right path.

Which leads me to the best boss I ever had. 

Imagine you're at work, managing a big project and people and faced with situations and decisions that are new to you. You have a one-on-one with your boss and you go in prepared to describe the situation and get insight and feedback on how to proceed. When you enter the office, your boss is nose to the laptop and barely acknowledges you're there. You know that you have limited time, so you ask if you should get started. "Yes, go ahead," your boss says, still not looking up from the laptop. 

You start describing the current status of and issues with the project. You describe what you've done so far to resolve issues as they've come up, and you end with the current dilemma and request advice on how to proceed. 

To which your boss replies, still not looking up, "what do you think you should do?"

Infuriating. If I knew what to do, I'd just do it! I want guidance, I want insight, I want feedback!

I had a couple of meetings that went exactly like this with my boss. Every time I left those meetings, I was pissed. How rude! What am I supposed to do? Why didn't my boss give me any advice?

And yet...

I was getting feedback. My boss was saying: "you don't need me. You actually know how to proceed. I trust you to figure this out. If you make a bad decision, that's ok, you'll come back to me and I'll repeat this again and you'll try something new, until you find the right answer."

It was these "non-feedback" meetings that gave me confidence to make decisions. They helped me learn to reflect, consider options and take my best guess. Sometimes I didn't make the best decision, but often I did. As I worked with my boss longer, our meetings became less about resolving issues and more about personal development and strategy. 

Coaches, mentors and managers can give helpful advice and guidance in some situations, but the best way they can be leveraged is to challenge you to do your own thinking and growing. When you're designing practice in immersive learning, consider designing with infuriating feedback: opportunities for reflection and safe failure. Not only will you build confidence in decision-making, but you'll be teaching leadership and reinforcing risk-taking and experimentation. 

How much do you know about immersive learning? Take the quiz!

I just presented a session on immersive learning at TechKnowledge in Las Vegas last week, and created this quiz as a companion piece.

How much do YOU know about immersive learning?

Note: I'm beta-testing a new quiz tool, so if you notice any bugs, drop me a note! I already notice that sometimes my feedback on each question isn't showing up... :)

Can't wait to see the results! If you like the quiz, buy the book!

Your success metrics aren't unique (or, why are we making analytics so hard?)

Last week I had the pleasure of attending TechKnowledge 2013, including giving a workshop on game design with Karl Kapp and presenting a concurrent session on immersive design. Two action-packed days in San Jose, for sure, and for once I got to see some of my esteemed colleagues' presentations as well.

I was most excited to see Ellen Wagner present on learning analytics and get a sneak peek into some of the work that she's been doing with the Gates Foundation. I was not disappointed; I loved hearing Ellen's practical advice to the packed room on how to approach analytics and not be intimidated by big data and the necessity of quantifying learning...or improvement, really...for our non-learning business colleagues. I appreciated her encouraging us to be able to frame the value that we bring to the organization in business terms without abandoning our expertise that qualifies us to take on an important organizational leadership role.

Watching Ellen's presentation left me thinking (which is normally a very good or very dangerous thing).

Every presentation that I've seen on analytics and big data end with the same call to action: "now, go back to your organization and figure out what you should be measuring."

Photo by WH "Snowflake" Bentley

Here's my challenge to you, my esteemed learning colleagues: shouldn't we all be measuring (basically) the same thing? We all suffer from the snowflake syndrome (I'm just like everyone else! I'm unique!), but to be fair, both statements are true. The real question is how much are we alike and how much are we different?

About ten years ago, I was working with a gentleman who had done an in-depth analysis across 15+ pharmaceutical companies, looking at what made their sales representatives successful. He found that 80% of the success indicators were exactly the same across all companies. The unique 20% related to the type of product, the therapeutic market the drug was being sold into, and the corporate culture differences among the companies.

Now, if you're a pharmaceutical company, where do you spend your analysis energy, knowing this? Right. Don't worry about identifying success markers for the 80%: use the identified metrics, provide training and support for those areas, and spend your analysis time identifying what the other 20% are that are unique to your company. EVERYONE should be measuring that 80%...that's baseline. Figuring out the 20% is what differentiates you from your competitors and supports your brand.

Could the same analysis be achieved for call center support? Project managers? Database engineers? Are there consistent success metrics that are common across industries?

If we could identify the success metrics across within or across industries, we wouldn't have to start from scratch to determine our analytics. We SHOULDN'T have to start from scratch. Don't we know what these things are? Can't we say, "Hey, you want to measure your organizations' success and performance improvement? Here is 80% of what you should be looking at, and what it means. Now, figuring out the 20% that's unique to your organization is up to you." (I'm not saying that it's literally 80% for every job function or every industry...but you get the idea.)

As learning professionals, we should know these benchmarks for our industries. If we don't, that is our very first starting point: find the "80%." What makes us the same? What makes us unique? Identify it, benchmark it, measure it. Then? Spend your time analyzing what makes you better than that 80%.

Your "assessment" is probably bullshit

Every year, the first morning of DevLearn, I usually wake up with some crazy epiphany about learning and training that sticks in my brain throughout the conference. This year the revelation was "assessment is meaningless." I decided to try to spend the conference (when I wasn't presenting or hosting the Emerging Tech stage) trying to find examples of meaningful assessment.

My mission was by and large futile.

Let me start by clarifying: for adult education and training, knowledge might be important for dinner party conversation, and it might even help you in a job interview, but more than likely what you know isn't a fair representation of what you can do. Learning objectives are very different than performance objectives.

Why are we still writing learning objectives and assessing for knowledge acquisition when the only thing the business cares about is what employees can do?

If you're an instructional designer and you're writing learning objectives, please stop. STOP. What you learned in your graduate program or by reading Dick & Carey is meaningless to business objectives and business leaders don't know what the hell ADDIE is. Executives don't care if employees passed all of their knowledge checks with 100%. The bottom line is not improved by multiple choice questions. WHAT YOU ARE MEASURING AS A BENCHMARK OF LEARNING IS MAKING INSTRUCTIONAL DESIGN OBSOLETE. Stop stroking your own ego. Stop trying to justify your decision to make a click-through elearning module or to drag the salesforce out of the field for two days for training by showing they all passed a multiple choice test. NO ONE BELIEVES YOU and NO ONE CARES (yes, I'm yelling).

Businesses care about performance. What are employees doing, and what do they need to do better, to improve business function. Is what you're teaching employees helping the business make more money or saving the business money? SHOW THAT. Use those words. Measure performance. Make what you do meaningful to your organization by setting performance objectives and measuring against those.  Provide performance support. Stop making the easy decisions. Make meaningful and relevant design choices.

If you can't assess something meaningful, it might be better to not assess anything at all.

/drops mic. walks off stage./

You can't measure learning, but you can measure behavior

I'm overstating it a bit with the title of this post, because sure, you can measure knowledge acquisition by pre-testing and post-testing, or iterative assessment. I know, I know...we can measure how much someone knows because we have standardized tests! (I really hope the sarcasm is evident in text...)

I spent the last three days at the mHealth Summit in Washington, DC and 19 hours manning the Ayogo booth, talking to amazingly interesting people about the potential of games to improve health outcomes. What mattered to everyone? It wasn't what people know...amazingly, most everyone actually knows what they need to do to be healthier. The challenge is to get people to actually DO those healthy things that will help them better manage their diabetes, reduce their risk for cardiovascular disease, etc.

When it comes to health, but really when it comes to ANYTHING, there is a knowing-doing gap. We all know this...so then why are we as a learning profession settling for assessing knowing? Knowing is not doing. The proof is in the behavior, and behavior can be easily measured.

We live in an age where everything we do is tracked. Do you carry a cell phone? Your wireless carrier knows where you take that phone all day, every day. Do you use a credit card or bank card? All of your purchases are tracked. Do you log onto the Internet? Every site that you visit is logged and recorded (yeah...I know...you delete the history. That just means your kids won't see those sites your visiting...but your Internet service provider still knows).

All of that data, and more...everything you post on Facebook, Twitter...everything you email...anything you do is trackable now. And more ways to track behavior are being created every day...sleep monitors, pedometers, glucose monitors...there is data EVERYWHERE and its all about you. And me. And the guy sitting in traffic next to you who's using his gps.

With all of this data, we can start making predictions about future outcomes. We can target specific communities or subsets of employees, populations, learners. We can provide information to the most relevant audiences in the most appropriate places.

As learning professionals, we should be thinking more closely about the implications of that data and what it means to know so much about a person's current status and the implications for her future status. Can we change the future? Why yes...yes we can. We can observe current behaviors, predict future outcomes, and use our expertise in learning and performance improvement to change behavior to improve those future outcomes.

We have access to so much behavioral data. How do we get people to change their behavior, when we know that people operate in a world of short-term benefit over long-term reward? We're not going to change those behaviors through knowledge training...we'll only change them through behaviorally-focused training. Games, simulations, contextualized practice...immersive learning environments are the bridge between having access to data and changing behavior for better results.

We can, and already do, measure behavior in almost every aspect of our lives. Learning professionals need to stop focusing on knowledge and start focusing on behavioral change as the basis of our design practice or risk obsolescence (see: Instructional Design is Dead). Our jobs aren't about making sure people know things...they are about making sure people can do things better. We can design those experiences and measure those outcomes. If we aren't doing that, we're not doing our jobs.

Gauntlet thrown.