Tuesday, April 24, 2012

A conversation on measurement and metrics (#ASTDLN)

These are my notes from the final workshop session at ASTD Learn Now in San Francisco.  Bob Mosher and Conrad Gottfredson of Ontuitive are leading a conversation on metrics and measurement. 

The things that we currently measure about learning are things that the learner doesn't really care about.  (% of course completions, pass/fail rate, # student days, etc.)

Bob Mosher -- there is value in smile sheets.  Research shows that if people like you're training, they'll take more.  There is defendable data when 12,000 people say that class was good.

When we measure the first two moments of learning need (new and more) we can measure with knowledge & skills gain -- certification, demonstrable skills, compliance.

Gloria Geary, Why Don't We Just Weigh Them? http://www.gwu.edu/~lto/gery.html

When we move to the world of performance support, we need to gather data to show that we make a difference to the organization to achieve its aims -- we need to measure competency and measure moments 3-5.  We need to tie it to on the job performance gains:

  • time to proficiency
  • lower support costs
  • completion of job-related tasks
  • increased user adoption
  • optimized business processes
  • customer/employee loyalty, morale, and/or retention
  • sales close/cycle time
But we need to be measuring what has a critical impact to the business.  We need to be measuring moments 3-5 (apply, solve, change)

There are three ways we can measure:
  • digital monitoring (we can track activity, see where they click, see where they spend time).
  • performer monitoring (quick checks)
  • “others” monitoring

We can do quick checks and ask the learner how they’re doing – but not for those things that would be catastrophic if we don’t do them right.

Shouldn’t the real measurement be whether or not they’re selling more chairs (assuming they’re selling chairs, of course)? What’s been the business impact?

Critical Skills Analysis – determine along a spectrum where things have critical impact to the business. Work with SMEs to create a rubric for the lines of business and for different skills.  
These are the lines of business perception of critical business actions – these aren’t the learning team’s perception…Bob and Con show a 1-7 ranked scale – a critical impact rating.  At one end is complete catastrophic results – e.g., someone will die.

Make sure you’re investing in measurement in the right places.  Figure out what’s happening in the right places.  Figure out what’s happening to improve performance support and to improve critical business impact.

We can’t measure everything. Don’t try to boil the ocean.  Measure what matters.  How deep do I go?

The new analytics:

Chad shares some data you can get from a mobile app:
  • time spend on a page
  • frequency of use
  • sharing info
  • type of info accessed
  • conversion points (are they doing what we’ve designed the experience to do?)
  • Other things…does access frequency go up or down over tie? Does engagement time go up and down?)
Analytics more in line with what marketing people look at.

Sample analytics that they got:
  • 20-25% of visits last between 10-30 mins (this was for a mobile quiz game that took about 2 mins – so people were spending more time here)
  • users returned to the app in less than one day
  • Game rules only comprised 1% of the time consumed – this was the manual/user guide – it confirmed for the developers that they had designed a good UI. 
This was data measured outside of the LMS. The digital analytic world – google analytics – it’s a new era in data. 

Yahoo Web Analytics – free tool used with advertisers.  To determine what % of business is coming from different channels.  At yahoo, using it to determine what learners are doing – what content they go to, what pages are useful, it allows them to understand behavior.  If there’s content out there that no one is looking at…then why?  This allows you to determine where they go and how long they stay – and to view it buy country/demographics.  Who’s using it?

Business example – health insurance provider using a performance support system:
  • 84% of sales force used the embedded learning solution DAILY
  • 6% increase in DAILY work productive – finding correct info, not waiting for answers, not bothering others (measurable, observable behaviors)
  • 2.4 hours saved per week per employee
  • So that means they had more time to sell.  $454K saved based on audience of 3,000 users
How would you go about gathering data at the moment of APPLY?

Chad Udell on Tools for Mobile Development (#ASTDLN)

Chad Udell (@visualrinse) of Float Learning is doing a quick rundown of tools he uses to help design, manage and build mobile projects.  This is day two of the ASTD Learn Now conference in San Francisco.

UI Stencils http://www.uistencils.com/ Use these templates and tools for rapid prototypes that are to scale for the device you're building to...

Project management tools like basecamp, assembla http://www.assembla.com/

Prototyping tools

  • Fieldtest - web based tool to create fast mobile prototypes. http://fieldtestapp.com/ (focused on smartphones and up = android, iOS, windows mobile)
    Chad shares a prorotype he created that you go checkout:  http://fldt.st/10d1c7f
  • App cooker http://appcooker.com/ An ipad app for building iphone and ipad apps (very meta) $25...
Once we've got the design prototypes...the tools to build the application:
Chad cannot recommend any of the rapid learning tools for dedicated mobile development.  

"Anything that's out there now, is not doing it well.  if you want to build high quality mobile learning apps, you have to use dedicated mobile tooling."

HTML, CSS, and learn some basic design for user experiences for mobile.

jQuery mobile  http://jquerymobile.com/ It's just like webdesign. Dreamweaver has this stuff built in for the newer versions.
(And Chad exhorts everyone to stop being scared of HTML - it's just an outline -- it's not programming!)

PhoneGap -- http://phonegap.com/ the only open source mobile framework that supports platforms.  Helps you bundle up content to an app that's distributable to a marketplace (android, apple, etc.) Adobe purchased phonegap and is building it into all of their tooling.

Misconception that apps have to go through the app store.  Apperian...helps you avoid putting proprietary content into the App store. www.apperian.com

Ah-ha’s and reflections from the first day of LearnNow (#ASTDLN)

The group shares some of their key moments and reflections from yesterday’s workshop session here at ASTD Learn Now conference in San Francisco.  I’m co-facilitating this conference with Bob Mosher, Conrad Gottfredson, and Chad Udell.

One project at a time.  We can start with one small piece and not try to do everything at once.

Thinking about blended learning vs. blended training.

The pyramid (see my notes from yesterday)

Two clicks, ten seconds.  We need to get people to the information they need quickly. 

The five moment of needs: new, more, apply, change, solve.  Moving the training org to think about all five of those moments and not just the first two.

We need to get to SUSTAIN.  Focus really on what’s needed.  All of the time expended on this long classes, etc. that people just forget after the event.

The implication for the formal learning if you change to thinking about this five moments of need – you re-design the formal event now to map into this new vision.

Monday, April 23, 2012

Implementing Social Learning (#ASTDLN)

My slides from my session at ASTD Learn Now in San Francisco.

Getting Started with Mobile Learning (Chad Udell) (#ASTDLN)

These are my live blogged notes from ASTD Learn Now Session with Chad Udell (@visualrinse) of Float Learning (www.floatlearning.com)

Mobile -- nearly pervasive devices.

Aspects of mobile learning:

  • mobility
  • ubiquity
  • accessibility
  • connectivity
  • context sensitivity (time, location and intent)
  • individuality (the content that I access is different than the content that you access)
Key questions you should be asking your organization before you get into mobile learning/key areas to consider:

What is driving mobile for you?  Build a business case.

Do you have a champion? Someone who's willing to wear the superhero suit?

Differences in design....instructional, interactive, contextual. You become more of a curator; you help people by putting wayside signs along the way. It's an aggregator role. You're competing with Angry Birds...

Lots of fragmentation - different operating systems, different devices.  Is your organization BYOD (bring your own device?) -- don't get excited about every new device coming in and freak out.

What about security? It can be a big deal and is often seen as a barrier to entry. You're going to have to become friends with the IS guys in your organization.

Mashups are the norm.  Data from one system (your CRM) may be living very close to other data (your company wiki). Your training content is now co-mingled and the lines are blurring.  We need to get comfortable with this new world.

Who are the stakeholders

Do you know your audience? Do you know their needs, their wants? Get them on board. How are they accessing data during the day on their company smart phones? Are they making calls? Texting? Going to your website?

Do you have a process? Strategy, design, develop, deliver. It needs to be agile, fast, rapid.

How do you and your company handle change? You're about to see a lot of it...

How can you evangelize mlearning within your org? Load up your ipad with ways you can help your business?  

Choosing an outside partner.  Focus on the content and outsource the tech.  Gaming app: takes HMTL5 with CSS3 compiled with PhoneGap. Scenario app: built with Objective C and iOS SDK....if that gives you the heebie jeebies, find someone who likes to do that!

What does success look like? Don't try to boil the ocean.  Make it simple...

The New Blend of Learning (#ASTDLN)

These are my live blogged notes from ASTD Learn Now Session with Conrad Gottfredson and Bob Mosher of Ontuitive (formerly LearningGuide Solutions). I am a co-facilitator for this 1.5 day conference, along with Bob, Conrad, and Chad Udell of Float Learning. About 55 learning colleagues gathered here at the lovely Hilton in San Francisco.

The "new blend" must be:

  • aligned
  • embedded (in the workflow and available at the moment of apply.  Two clicks, ten seconds.  Needs to be close to the moment and instant...don't want lengthy responses if it needs to be immediate at the moment of apply. I need to transfer that information immediately at the moment of application.  Sharepoint and LMSs can get it to you, but it's not immediate -- the content is too long.)
  • contextual (according to circumstance, specific roles and varying access needs. Anticipate the key and critical contexts: moment of need, job role, work flow, content access requirements, circumstance.  An instance could be different than a specific workflow -- e.g., a teacher has to respond to an angry parent. That's an instance...not something that happens in a specific place in a workflow. Or the instance of an upset customer.)
  • just enough (only what's needed in the form needed to effectively perform inside the business process. They pyramid's the key.)
  • integrated (with formal learning to compliment and extend current learning investments)

The training wheel metaphor:
  1. metacognition
  2. cognitive apprenticeship
In your training/formal class: build in an evolutionary methodology of trust, comfort and belief.  Start a training program off with cases...over the formal class pull away the training wheels -- trainer walks them through it, solve a problem with a peer, and so by the end the individual is solving the problem on their own.

The Five Moments of Learning Need (#ASTDLN)

These are my live blogged notes from ASTD Learn Now Session with Conrad Gottfredson and Bob Mosher of Ontuitive (formerly LearningGuide Solutions). I am a co-facilitator for this 1.5 day conference, along with Bob, Conrad, and Chad Udell of Float Learning. About 55 learning colleagues gathered here at the lovely Hilton in San Francisco.

“I don’t care what they know, I do care what they do.” 

It’s all about APPLY.

Training is good and New and More. Getting them started, keeping them going.

Need to help them CHANGE and SOLVE.  This is part of the APPLY world.

Throw out the word “informal learning” – try to get the CFO to pay for informal learning. Instead, let’s shift to the word “performer support” – that’s a defined, measurable word.  

Note that it’s “performer” and not “performance” – it’s more human.  Performance support is the enabler, the solution that we build.

“Performance is reality. Forget everything else.” Harold S. Geneen CEO of ITT
In today’s dynamic world, I often need to learn at that moment of APPLY. Apply is time sensitive – I can’t get out and hunt down something in an LMS right now…

The new challenge of learning = knowledge aggregation. In today’s world we’ve got too much knowledge and info.  We’ve got access to more than enough stuff.  We live in a world of abundance. (In the olden days, you needed the expert up there to get it out to the people).  We have been in a pile on mode for 20 years.

We need to support the learner DURING, BEFORE and AFTER…

We can’t survive in a world of “I’m going to help you when you’re in the problem…”  -- that shouldn’t be all they have.  We need to prepare them…

Allison Rossett: “If we were better at before and after, we’d have fewer durings.”

Which of these five moments of need does your organization address thoroughly?
  1. New
  2. More
  3. Apply
  4. Change
  5. Solve

We have been blending training, not blending learning.

Most of us have structural frameworks to do #1 and #2…we haven’t had methodologies for 3, 4, and 5….

The pyramid – the learner sits at the top at the moment of need. 

At the top: process, steps, concepts.

Sometimes that’s not enough and they want resources: references, learning, people (this is not good to bad hierarchy)

(Don’t confuse modality with design.)

(2 clicks, 10 seconds -- people will take bad stuff fast over good stuff that's hard to find.  We need to get people to the right things quickly).

In your class on the new software program -- teach them what it can do, talk them through scenarios. Don't show all the steps and the clicks and the drop-down menus.  Show the steps in an EPSS...

Bob now demos a LearningGuide built into a Salesforce.com implementation.  On a page, you click a LearningGuide button -- it brings up a whole list or links and info for the entire pyramid.  It's context sensitive.

Can do this same thing for a soft skill process -- Bob shows a graphical dashboard for a job requisition process.  Each step of the hiring process is shown and you can click down to find the steps/tasks/training resources (they pyramid) etc.

This type of system can sit on top of an LMS. So you can do tracking.

We overteach.  Let's stop doing that.  We don't have the time anymore.

The Changing Nature of Business & Its Impact on Learning (#ASTDLN)

These are my live blogged notes from ASTD Learn Now Session with Conrad Gottfredson and Bob Mosher of Ontuitive (formerly LearningGuide Solutions). I am a co-facilitator for this 1.5 day conference, along with Bob, Conrad, and Chad Udell of Float Learning. About 55 learning colleagues gathered here at the lovely Hilton in San Francisco.
  • Economic disruption can happen in one part of the world and carries its impact across the world…disrupting our businesses.
  • Technological disruption
  • Demographic churn
  • Political instability (the economic consequences of terrorism and upheaval)
The challenge for organizations: to learn at the speed of change.  We must continuously undergo new skill cycles to prepare for new competitive cycles.  Who hear can really plan for three years from now? Who can say, we’ve got our learning solutions in place? It’s like we’re all trying to change tires on moving trucks.

So what is the new normal? How do we learn at the speed of change?

“The consequences of standing still look worse than the consequences of taking a chance on change.” Denis Pombriant

3 fundamental functions that have to work together within an org to help adapt:
  • intelligence function
  • strategy function
  • learning function (we have to learn from strategy and give it back)
Today we need to be dynamic learners:
  • 1957-1981 = Permanent Learning – one-time learning for permanent qualification
  • 1981-2004 = Continuous Learning – for ongoing qualification
  • 2004-2009 = Dynamic Learning – rapid, unlearning, relearning, collaborative, self-directed learning at the moment of need
The nature of the leader in the new normal….from leader as expert to leader as learner….personal credibility based on personal learning agility…it’s a different set of leadership skills.

If you want to live in LearnNow…you have to live in the world of SUSTAIN. It’s not training anymore. We can’t keep a classroom mentality. We need to live in sustain and transfer. Get out of the brick and mortar or the LMS.

For a new learner, training is still the right starting place. We don’t advocate cutting training. We do advocate changing it A LOT.  Training helps you get to mastery. But mastery does not get you to sustain. Knowing stuff does not mean doing stuff. 

The LMS plays in the red zone = training. 

To get to sustain, we need to embed performance support across this whole spectrum.

In the sustain stage, what if people could keep raising their own bar…innovating, maintaining mastery, continuing to learn while getting better?  LearnNow techs including EPSS (embedded performance support solution – not electronic…, social and mobile help this.