About the Biz, Methodologies

Truth in User Testing

User Testing

Back in ’98, like many rookie entrepreneurs starting their first company, all I cared about was building the most super fantastic product the world has ever seen.  And we did, we built a great product.  There was just one problem, nobody wanted it.  To be fair, there were external factors that played into the low sales (9/11 being one of ’em) but those are all just excuses.  We didn’t spend nearly enough time talking to potential users.  I have since seen this happen again and again and again in businesses that I have consulted with and some that I’ve worked for.

Talking to your customers before and during the building phases of a product seems like an obvious step.  So why don’t companies do this more often?  Turns out, there are a lot of reasons.  I’m not saying any of these reasons are good ones, but they are worth looking at because they do provide a window into the human condition.  Here’s a short list that could easily grow to 10+ pages:

  • It’s hard.
    • Finding prospects in your target demographic that are willing to give you the time is not easy.
    • True user testing is not a one-time thing.  You need to keep asking for feedback and it’s going to be hard every time.
  • It’s expensive
    • Most of the time you have to incent these potential customers to try the product and give real feedback.
  • It’s emotionally taxing
    • You have to put your baby out there.  Nobody likes to hear their baby is ugly.
    • It’s much easier to believe in the narrative that you will be that 1 in a million that gets it right the first time.
  • You might not like what you hear
    • Truth is always a double edged sword.  You’re not going to like everything you hear but you are almost guaranteed to learn something.
    • The irrational fear that all the feedback will be negative.  Wouldn’t you rather find out now so you can pivot?

That’s just the tip of the iceberg.  After vowing never to make the no user testing mistake again, we decided that we are going to test this idea early and often.  As I’ve aged and mellowed a bit, like a fine bottle of Mad Dog 20/20, I’ve learned that you are always going to be your own harshest critic.  Most of the feedback you get will not be nearly as bad as the insults hurled like feces by your internal lizard brain monkey.  The best thing you can do is embrace it.  Treat all the barbs and arrows of negative feedback as an initiation to greatness.  If you can shift your mindset to treat every piece of feedback as a gift that will enhance your learning, you really start to look forward to these things.

Our first round of testing

That brings us to our first round of testing.  We had guinea pig rounds before this to iron out a lot of the little things but this was to be our first real test.  This took place on Saturday night with a small group of test subjects.  We plied them with food and booze to make them compliant and we got to work.  We took them through the entire process.  This is still the paper process as we are not building any software until we are sure this has a need in the market.  Before we started, they had homework to do all the ‘Know Thyself’ steps.  This is the Meyers-Briggs, the 5 Love Languages and the Character Strengths and Virtues.  This put the working group in a pretty introspective state when they arrived.  This was a great move because everyone was excited about the results of their tests and interested to see what was next.  They really wanted to know how we were going to drive a plan and a purpose from these next steps.

We had all the steps in a time boxed agenda.  This was the first chink in the armor.  Folks didn’t like being timed while thinking about these sweeping, grand life issues.  Getting through the entire process was important though, so we stuck to it.  We would have considered the testing a failure if we did not walk out of this session with personal scorecards.  Some of the steps that we thought would go really quickly took forever and some that looked like they would take a long time, went really quickly.  We will modify the agenda to account for these time discrepancies before our next round of user testing.

As we went from skills to scorecard, we heard another set of concerns.  The first was that it felt a little too ‘business-y’.  Not surprising considering our backgrounds.  The second was the concern that we wouldn’t be able to easily turn this into software.  The argument was that without someone mediating or coaching the process, it would be very difficult to train the end user in building goals that were actionable.  Taking care of the ‘business-y’ feel is a lot simpler of a problem to solve.  We can get rid of all the acronyms and make the jargon much more approachable.  This is a fixable user experience issue.

The problem of how we do this without a coach or mediator is much more difficult.  This led us to brainstorm some ideas.  The first was that we could intersperse a bunch of videos in between steps and treat it more like an online educator’s approach to a  personal development plan.  Something like what you would find on Udemy.  We could also have our primary target be life coaches and just have the software enhance or document the improvements that they are already making with their clients.  The brainstorm I liked the most though was to ease the users into it.  Instead of starting with building out a full 3 month scorecard, you start with one easily surmountable problem to solve.  You use the same methodologies to solve this problem but you train the user on the lingo and make it a lot less overwhelming at the start.  Someone even compared it to Home Advisor’s site.  You pick the problem you are trying to solve, get access to the literature and ultimately a contractor who can help you through it if you can’t do it yourself.

This reminded me a bit of how many video games are approached.  You always start off on a newbie level which introduces you to the basic controls.  the game then has you find your first quest or mission.  Once you complete that and have a handle on how to navigate through the world, the challenges get tougher and tougher.  A lot like life.  We have talked about gamifying the software from the get go, so this is not a new idea.  Introducing gamification at the very beginning of the process is new however.  The gamification would drive the learning and the scorecard.  We haven’t made a decision on this as we still need more testing but this idea feels like we’re getting warmer.

Folks also struggled with building out their metrics.  This is a very common problem in our business backgrounds.  Understanding what to measure that will actually affect change is always a trial and error process.  The concern is that we will lose our users if this trial and error process is too haphazard and they run into too many failures early on.  This concern also lends itself to a far more curated problem to solve for your introduction to the software.  Once the end user sees how this works, we can then unlock the functionality that allows them to determine their own metrics.

Even though there were plenty of bumps in the process, all of our test subjects came out of the meeting super charged up.  They also all walked out with three month scorecards.  There was a ton of energy in the room and the feedback session at the end went on for almost an hour when we expected only about ten minutes of feedback.  This felt like we were on to something.  Boredom is the worst possible response and we were far from that.  People have a ton of emotion about the subject and a lot of hope that things can be better.  That seems worth pursuing.

Next Steps

We have three more user testing sessions lined up for this first iteration of the process.  It’s tempting to make major process modification after the first round of feedback but we do want to get feedback from different age groups and other diverse demographic variables.   We will be making small tweaks to the process to make it run smoother but we will not be shifting any of the fundamental process steps yet.

The other step we will be taking is in monitoring the progress of these scorecards.  Building the scorecard is only step one in the process.  The next step is making sure folks follow through on their plans.  This first round of monitoring will teach us a lot about how folks execute without job based pressures to do so.  We have a lot of techniques that we will be employing to motivate folks to follow through.  It will be interesting to find out which work the best!

Stay tuned for our next process blog!

Leave a Reply