Data is meaningless out of context: thoughts on the Tin Can API (xAPI)

Just posted a comment on Dr. Eric Fox's blog post from last fall on his thoughts on the Tin Can API:  A Behavioral Scientist's Initial Thoughts on the Tin Can API, Big Data and Learning Analytics. If you haven't read it, you should. My comment to him is re-posted below. Let's keep the conversation going. 

Data is meaningless out of context

Hi Eric,

GREAT post and happy to see you've delved into one of my biggest issues with the Tin Can API. Ironically, someone pointed out this post to me, having just left ASTD's TechKnowledge conference where there was continued buzz and interest. As a immersive learning designer and someone who has struggled with how to quantify practice into meaningful data, I am thrilled that there maybe an emerging standard that could help capture that data. The problem, as you correctly point out, is that reporting activity neither demonstrates learning nor performance improvement.

The hard work is in establishing actual performance metrics and measuring improvement through various learning activities. Simply reporting that you did something doesn't show qualitatively OR quantitatively whether that activity has any impact on what you know, or what you can do better.

I see potential here, but it troubles me that people are too "oooh! shiny!" about one minor piece of a much bigger piece of work, namely, correlating activity to performance. There are A LOT of potential problems with the Tin Can API, and so far, I haven't seen any best practices, use cases, or case studies that demonstrate its most effective use. Another problem? Those directly involved in the creation of the Tin Can API jumping into every discussion and squashing the much needed conversation among the rest of the community. Much like a community manager who tries to force conversations in a certain direction, there has been much talk in the industry about our inability to have deep conversations about the pros and cons of the Tin Can API without the developers and evangelists inserting themselves and trying to guide the conversations. Like pushy sales people, they are becoming a turn off for people who want to research, investigate and share their own conclusions. If I want to hear the sales pitch, I'll talk to a sales person. If I want to talk to my peers, I go to social media. Unfortunately, the sales people have hijacked these social conversations and are creating an atmosphere where no one wants to participate.

It reminds me too keenly of my experience with virtual worlds...the developers so in love with what they had built that they stopped listening to the concerns, needs, and objections of consumers and their potential customers. When you build something, you sometimes get too close to it to be able to see the chinks in the armor. When developers start arguing with naysayers, it's a clear sign that it's time for them to let go. Like an artist, you can have intention with your work, but the true meaning is what every person brings to it. It may be time for the developers to step down, and let practitioners lead the next phase of the discussion to help the Tin Can API survive its inevitable fall into the "trough of disillusionment."

Let's hope the conversation continues and that the real problems with the Tin Can API are not ignored. No system is perfect, but ignoring the problems, or trying to squash the conversation about them, does not make them go away. I'm hopeful that we can all learn from each other in making the Tin Can API useful and meaningful.

A love story for Valentine's Day

When I explain my current family situation to, well, anyone, I get looks of wonder, a clear feeling of "I don't know how you do that," and sometimes even pity.

Let me tell you: My life is non-traditionally glorious because it's full of love.

Love comes and goes in your life. Family, friends...it's amazing how we all evolve together or grow apart. When I say that the last few years have been the most challenging of my life, it's because these last five years have challenged every belief I ever held about love and made me see who I really am.

Several years ago, I made a series of bad choices. Some were clearly my fault, some were guided by broken promises. My marriage to a wonderful man was failing because it turned out that the life we had built together wasn't making either of us very happy. I met someone else who mirrored every hurt I was feeling and who said all the right things to me. The timing wasn't perfect, but I told myself, "when is it ever, really?" As I ended one relationship and another was beginning, I saw an end to my loneliness in sight.

Words are easy. I was so naive, so blind in my faith in his words that I let myself circle deeper and deeper into a situation that could never end happily. The lies that he perpetuated in every aspect of his life, that I knew and accepted, only foreshadowed the depth of the lies he was telling me too. I thought I was too smart to be deceived, but I believed everything he told me. Why? Because I wanted to. I wanted to believe the pictures he painted for me with his words, I wanted to believe in the future he described, because the present was miserable and lonely. Having faith in him was directly tied to my faith that life would get better.

Over time, I lost more and more of myself and accepted smaller and smaller pieces of happiness. The final straw: a holiday weekend where he texted and called me with detailed descriptions of what he was doing at home alone, all the while he was away for the weekend and not alone. When I confronted him, he blamed me, told me that I didn't want to know the truth.

Actually? Yes, I did.

I hold myself to unachievable standards in all things, and usually that makes me spend a good part of each day feeling like a failure. It's not the rational part: I see the things I've accomplished, but more often I focus on the things that I haven't done, that I screwed up or outright failed at. There's lots of those, for sure. But this failure, this acceptance of lies and broken promises and holding on so tightly to one path to happiness, this failure broke me. It broke my heart, it broke my faith in myself, and it broke my trust in words.

So far, this isn't much of a love story, huh?

When I was at my lowest, I took a risk. I went on a date. Worse, I went on a blind date with someone I had only met online. That date could have made for a really funny anecdote of bad chemistry and deal-breaker personality quirks. Instead, it led me here, to the happiest I've ever been.

When you have nothing to lose is when you're the most honest. I wasn't looking for a relationship; I don't really know that I was looking for anything more than getting out of the house and having a drink, to be honest. On our first date, I'm pretty sure I said anything I could think of to try to scare John off. My divorce wasn't final. My ex lived in the apartment in the lower level of my house. I have three kids. I was still shaking free from that horribly toxic relationship. No matter what I threw at him, he laughed at me, somewhat incredulous that we shared such similar paths. He had his own stories, and his own three kids. We joked that I was probably a crazy cat lady, and that he was just uncovering more of my cats. We talked for hours, closed the restaurant down, and left each other awkwardly in the parking lot that night without a good night kiss.

It's hard to believe that was only a little over a year ago. In the months that followed, we stayed honest about our tender but healing wounds. He did his best not to scare me off with too much attention and I did my best not to run away to spare him all of my complications. It seemed too easy between the two of us. Friendship, laughter, excellent adventures...and honesty. He always kept his word. He's got an infectious laugh and killer dimples. It took me a little time to see myself as he seems to see me. It's taken even longer for me to forgive myself for my past mistakes, no matter how warmly he embraced me and accepted them as part of who I was and who I am. Sometimes I still need him to remind me.

Oh yeah, we also got tattoos...

In a whirlwind year, we moved across the country, finalized our divorces, bought a house, started new jobs and integrated our kids. My ex moved here, found a new job, a new house...he's part of our crazy blended family too. Six kids, two dogs, my ex, John and I.

I get it. I know why people look at me in horrified amazement.

I know these things: I am loved. I have a partner who is true in all things and who I can depend on for anything. He is my constant reminder of what good looks like. I appreciate him not just for what he does but for who he is. I have an amazing mix of kids who each carry pieces of my heart and who teach me and help me grow as a mother and as a person. I laugh every day. I know what is important. I will never settle for anything less than ecstatically happy ever again.

It's an amazing adventure and I can't wait to read the next chapter in this love story.

Your success metrics aren't unique (or, why are we making analytics so hard?)

Last week I had the pleasure of attending TechKnowledge 2013, including giving a workshop on game design with Karl Kapp and presenting a concurrent session on immersive design. Two action-packed days in San Jose, for sure, and for once I got to see some of my esteemed colleagues' presentations as well.

I was most excited to see Ellen Wagner present on learning analytics and get a sneak peek into some of the work that she's been doing with the Gates Foundation. I was not disappointed; I loved hearing Ellen's practical advice to the packed room on how to approach analytics and not be intimidated by big data and the necessity of quantifying learning...or improvement, really...for our non-learning business colleagues. I appreciated her encouraging us to be able to frame the value that we bring to the organization in business terms without abandoning our expertise that qualifies us to take on an important organizational leadership role.

Watching Ellen's presentation left me thinking (which is normally a very good or very dangerous thing).

Every presentation that I've seen on analytics and big data end with the same call to action: "now, go back to your organization and figure out what you should be measuring."

Photo by WH "Snowflake" Bentley

Here's my challenge to you, my esteemed learning colleagues: shouldn't we all be measuring (basically) the same thing? We all suffer from the snowflake syndrome (I'm just like everyone else! I'm unique!), but to be fair, both statements are true. The real question is how much are we alike and how much are we different?

About ten years ago, I was working with a gentleman who had done an in-depth analysis across 15+ pharmaceutical companies, looking at what made their sales representatives successful. He found that 80% of the success indicators were exactly the same across all companies. The unique 20% related to the type of product, the therapeutic market the drug was being sold into, and the corporate culture differences among the companies.

Now, if you're a pharmaceutical company, where do you spend your analysis energy, knowing this? Right. Don't worry about identifying success markers for the 80%: use the identified metrics, provide training and support for those areas, and spend your analysis time identifying what the other 20% are that are unique to your company. EVERYONE should be measuring that 80%...that's baseline. Figuring out the 20% is what differentiates you from your competitors and supports your brand.

Could the same analysis be achieved for call center support? Project managers? Database engineers? Are there consistent success metrics that are common across industries?

If we could identify the success metrics across within or across industries, we wouldn't have to start from scratch to determine our analytics. We SHOULDN'T have to start from scratch. Don't we know what these things are? Can't we say, "Hey, you want to measure your organizations' success and performance improvement? Here is 80% of what you should be looking at, and what it means. Now, figuring out the 20% that's unique to your organization is up to you." (I'm not saying that it's literally 80% for every job function or every industry...but you get the idea.)

As learning professionals, we should know these benchmarks for our industries. If we don't, that is our very first starting point: find the "80%." What makes us the same? What makes us unique? Identify it, benchmark it, measure it. Then? Spend your time analyzing what makes you better than that 80%.