usability testing: the process of learning about users from users by observing them using a product to accomplish some specific goals of interest to them, from Barnum (2010), page 6
usability: the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use, from ISO 9241-11, quoted in Barnum (2010), page 11
Recall Buxton’s characteristics
Here are two books: one is for amateur usability testers and one is for professional usability testers. (You can still be a UX professional and be an amateur usability tester. They catch many problems.)
How many problems can you catch? As many as you can fix, plus any that don’t matter
Most professionals of my acquaintance have read the amateur’s book, by the way. Maybe they want to be sure that they are not missing anything an amateur would catch.
The book is called Rocket surgery made easy, the do-it-yourself guide to finding and fixing usability problems by Steve Krug, 2010.
Krug is best known for the book Don’t Make Me Think, Krug (2005). I’ve always resisted reading that book because the title gives a questionable command.
I was taught the goodness of thinking as an axiom.
Even test a similar product before you’ve done any design work at all!
It’s more important to test frequently than to get the right participants or more participants.
Seeing is believing. Observing makes you realize how different users are from you. More observers are better.
How many experience the problem and how severe is it for those who do?
Do something and don’t try to do everything. Tweak, don’t redesign. Take something away.
Make a list of tasks!
Make each task into a scenario.
The scenario adds context, e.g., you are …, you need …, and supplies information, e.g., username and password, but doesn’t give clues!
Pilot test the scenarios.
〈 pause to watch a Steve Krug usability test 〉
Usability testing essentials: ready, set… test! by Carol Barnum, 2010.
After you digest the amateur’s book, it’s time to tackle this one, especially chapters 5 through 7, describing planning, preparing, and conducting a test.
Figure 3.1 from Barnum (2010) p 55 shows the user centered design process in which usability testing is embedded.
Consider each of these in turn …
(Remember to arrange for a backup participant)
(measure anything you can quantify that might correlate with success)
As a well-researched instrument, plenty is known about what it means. For example, the average score for 19 studies reported by Lewis (2009) was 62.1 rather than the 50/100 that might be expected at a glance. Another study (Bangor, 2008) reported an average score even higher, at 70/100.
The source for this material is a book called Research methods in hci
Following are eight heuristics known as Shneiderman’s Golden Rules of Interface Design, Shneiderman (2017)
Automated testing isoften used for WCAG checking (web content accessibility guidelines)
This is what people think of when they think of usability testing
Here are two views of the testing process
Rubin and Chisnell (2008) proposed eight steps in their Handbook of Usability Testing
Lazar (2006) also proposed eight steps in the book Web Usability
(Nielsen and Landauer, 1993)
(Nielsen and Landauer, 1993)
(Hwang and Salvendy, 2010)
… for the following reasons
(Lindgaard and Chattratichart, 2007)
Instead, number of flaws found will depend on design and scope of tasks!
Do you need one representative from each group? or five? or ten?
Children alone are less likely to criticize than children in pairs
Do you need recordings?
Why not test users you can not meet?
e.g., Amazon’s mechanical turk
You usually need a list of tasks for the participant to perform
One study found participants had to be paid more for tasks involving own financial data
Plan for whether interventions are to be allowed and, if so, how they’ll work. Should they only be for insurmountable obstacles?
Or a reflection session to get verbal feedback in case think aloud would interfere with task performance
People don’t show up unless you send reminders
People show up late or get interrupted, so you have to allow extra time
May have an irb (Institutional Review Board) protocol but should have some protocol anyway
The participant should be reminded of this because it will otherwise appear that the user is being tested
The protocol for the session should be followed but can be looser than for an experiment
Why would you even care about Morae? It’s obsolete and discontinued. But … some people still use it and the workflow is quite good even if you don’t use it. So let’s explore it from the workflow standpoint.
Think about the kind of session you’ll run
If you’re testing a paper prototype, you probably need both cameras. You can set up Morae in paper prototype mode so that it does not record the PC screen.
You can set up Morae in moderated usability test mode in a lab and run the test solo or with observers in the control room.
This is the most likely setup.
Don’t discount the value of autopilot. This is meant for unmoderated sessions but you can use it to free you from having to juggle too many things to remember during the session. The only caveat is that you have to set it up properly or you will wind up abandoning it and returning to moderated mode.
Now it is time get some evidence from your recordings and prepare it for presentation.
You’ll want to find video clips that illustrate important points.
You’ll want to log markers so you can get a graph that supports your findings.
caveat: This is about academic writing and industrial writing is usually less strict
Did we measure what we said we measured?
… but should it really end there?
more design ideas
At least, not necessarily
Understand participants better first!
… before you try to make design choices
ask why? three times
to get beneath the surface
until you understand the usability problems
when you’re ready to make design choices
This slideshow was produced using
Fonts are League Gothic and Lato