Last week we conducted a substantial usability test evolution for Reacht.  We were seeking truth about several things:

• problem-solution fit
• perception of the unique selling proposition (USP)
• application discoverability
• usability
• colors, branding, style

We got the truth 🙂  Some of it was a pleasure to receive, some not  :).  All was valuable.  I want to reiterate my professional and personal thanks to those that helped us with this important task!  Will Fulton, our Product Manager, did most of the work.  If you have specific questions be sure to email him.

Volumes have been written about usability testing by people far better qualified than I.  A few useful links are shared below:

This post doesn’t intend to make you smarter about usability testing in general.  Rather, you’ll learn what we learned from our first (of many!) usability testing evolutions.  Would love to get your feedback @rspz!

Group Size

We had twelve (12) participants in our usability test.  It was great to see that many people interested in providing us feedback.

That said, we will recruit seven (7) participants for our next usability test with the idea that five (5) will participate.  After each test proctor had worked with three (3) people the variety in the feedback dropped.  That is, it took three (3) usability tests for the proctor to be able to identify The Most Important Thing we need to be working on.

Philosophically, we are likely to conduct more frequent smaller usability tests early in product development.

Qualitative Feedback

Early in a product’s life many decisions remain to be made.  The most important, from a product perspective, is what to work on next.  This doesn’t require large values of N or days-long usability tests.  For the moment we’ll be sticking to gathering qualitative feedback from our usability tests.

This doesn’t mean we don’t believe in “If you don’t measure it, you can’t improve it.”  We do!

Think aloud, think aloud, think aloud

The (qualitative) value in our usability test was in learning what our test participants were thinking.  Repeatedly encourage everyone to THINK ALOUD during the tests!

Recruiting

David Malaxos “stood” in front of the elevators (on our Double Robotics robot) and recruited some ad hoc participants.  This was an excellent way to foment interest and curiosity and engage with our fellow building colleagues.  This is a definite repeat.  David took a nice “meta selfie” to document the tactic.

Video

We recorded each interview from two perspectives:

• tight device-only camera angle
• wider angle for including the participant and the test proctor

Best practice suggests that video is a good reference when the team needs to more carefully interpret a participant’s behavior (e.g., was the participant thinking or expressing frustration).

Our pursuit of large format learning tells me we won’t need video for a little bit.  I think we’ll either (a) skip video or (b) rely on a single tightly-composed device shot during out next test evolution.

Scenario composition

We had a mismatch between the words in the software and the words (see Power of Words next) in the usability scenarios.  Phrases like “Add a new member” and “Join code” were intended to be compatible but were demonstrated to be incompatible.

The power of words

We dropped “Join code” in favor of “Group code” in the user interface.  It’s a simple change that is much more in keeping with how we intend users to behave and think about the groups and group segments in which they are a member and to which they are sending actionable messages.

The word “join” connotes and action and, more specifically, a non-member action (i.e.,  non-member JOINS a group).  In hindsight, this isn’t at all what we wanted to communicate.  This became painfully obvious by the end of only the second interview that evening.

Discoverability

It’s dangerously easy to lose site of your first-time user.  As you incept, elaborate, refine, design, implement, QA, test and release software your “feature virginity” is lost to inevitable expertise and familiarity.  Given how much distraction our customers/product users face it’s a survival issue for us to preserve a first-time user perspective in order that our user acquisition not suffer.

The usability test is the KEY component of ensuring that first-time users understand and enjoy the product.  We’re betting that if we do that user acquisition and conversion will be off to a good start.

Bryan Wengren (@bwengren) observes a usability test being done by Chris Muldrow (@muldrow).

Lots of lessons learned and we can’t wait to apply them and make Reacht better!