definition

usability testing: the process of learning about users from users by observing them using a product to accomplish some specific goals of interest to them, from Barnum (2010), page 6

definition

usability: the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use, from ISO 9241-11, quoted in Barnum (2010), page 11

formative

summative

We’ve discussed lofi prototypes

Recall Buxton’s characteristics

Lofi purpose drives lofi testing

How to initiate a conversation?

Encourage think-aloud

Watch the response

Don’t be afraid of pauses

Put the participant at ease

Aids

how serious are you?

Here are two books: one is for amateur usability testers and one is for professional usability testers. (You can still be a UX professional and be an amateur usability tester. They catch many problems.)

How many problems can you catch? As many as you can fix, plus any that don’t matter

first, the amateur’s book

Most professionals of my acquaintance have read the amateur’s book, by the way. Maybe they want to be sure that they are not missing anything an amateur would catch.

The book is called Rocket surgery made easy, the do-it-yourself guide to finding and fixing usability problems by Steve Krug, 2010.

Steve Krug

Krug is best known for the book Don’t Make Me Think, Krug (2005). I’ve always resisted reading that book because the title gives a questionable command.

I was taught the goodness of thinking as an axiom.

Steve Krug’s maxims

1. Start earlier than you think makes sense
2. A morning a month, that’s all we ask
3. Recruit loosely and grade on a curve
4. Make it a spectator sport
5. Focus ruthlessly on a small number of the most important problems
6. When fixing problems, always do the least you can do

A morning a month, that’s all we ask.

Start earlier than you think makes sense.

Even test a similar product before you’ve done any design work at all!

Recruit loosely and grade on a curve.

It’s more important to test frequently than to get the right participants or more participants.

Make it a spectator sport.

Seeing is believing. Observing makes you realize how different users are from you. More observers are better.

Focus ruthlessly on a small number of the most important problems.

How many experience the problem and how severe is it for those who do?

When fixing problems, always do the least you can do.

Do something and don’t try to do everything. Tweak, don’t redesign. Take something away.

Krug’s common problems

task-level measurements

Make a list of tasks!

Make each task into a scenario.

The scenario adds context, e.g., you are …, you need …, and supplies information, e.g., username and password, but doesn’t give clues!

Pilot test the scenarios.

https://youtu.be/VTW1yYUqBm8

the professional’s book

Usability testing essentials: ready, set… test! by Carol Barnum, 2010.

After you digest the amateur’s book, it’s time to tackle this one, especially chapters 5 through 7, describing planning, preparing, and conducting a test.

ideas from the professional’s book

small studies

Figure 3.1 from Barnum (2010) p 55 shows the user centered design process in which usability testing is embedded.

user-centered design according to Barnum (2010)

process according to Barnum (2010)

Consider each of these in turn …

planning for usability testing (Barnum, 2010, ch 5)

preparing for usability testing (Barnum, 2010, ch 6)

(Remember to arrange for a backup participant)

conducting a usability test (Barnum, 2010, ch 7)

summative testing, according to Barnum

metrics

quantitative measurements

(measure anything you can quantify that might correlate with success)

system usability scale (SUS)

As a well-researched instrument, plenty is known about what it means. For example, the average score for 19 studies reported by Lewis (2009) was 62.1 rather than the 50/100 that might be expected at a glance. Another study (Bangor, 2008) reported an average score even higher, at 70/100.

tools

resources

http://rocketsurgerymadeeasy.com

http://usability.gov

http://uxpa.org

http://credibility.stanford.edu/guidelines/index.html

Lazar, Feng, and Hochheiser (2017)

The source for this material is a book called Research methods in hci

what is being tested?

goals

differences from academic research

success of usability testing

process drivers

changing treatment during process

observation differs from ethnography

categories

expert-based testing

expert review types

shneiderman’s heuristics:

Following are eight heuristics known as Shneiderman’s Golden Rules of Interface Design, Shneiderman (2017)

Eight golden rules

automated testing

Automated testing isoften used for WCAG checking (web content accessibility guidelines)

user-based testing

This is what people think of when they think of usability testing

validation testing

stages of testing

Here are two views of the testing process

First view

Rubin and Chisnell (2008) proposed eight steps in their Handbook of Usability Testing

Eight stages

Second view

Lazar (2006) also proposed eight steps in the book Web Usability

Lazar’s stages

five

(Virzi, 1992)

no, seven or fifteen

(Nielsen and Landauer, 1993)

three point two!

(Nielsen and Landauer, 1993)

no, ten

(Hwang and Salvendy, 2010)

no, twelve

(Caine, 2016)

three

(Krug, no date)

all numbers are correct

(Lewis, 2006)

… for the following reasons

Choosing the number

numbers don’t matter

(Lindgaard and Chattratichart, 2007)

Instead, number of flaws found will depend on design and scope of tasks!

testing groups

Do you need one representative from each group? or five? or ten?

testing pairs

Children alone are less likely to criticize than children in pairs

More on numbers

The final word on numbers: five

(folklore)

what do you need?

Do you need recordings?

remote testing

Why not test users you can not meet?

e.g., Amazon’s Mechanical Turk (MTurk), ClickWorker, crowdsourcing platforms for micro-tasks

pros of mechanical turk

cons of mechanical turk

user nights at yahoo!

task lists

You usually need a list of tasks for the participant to perform

qualities of the task list

workload

One study found participants had to be paid more for tasks involving own financial data

navigation

The task list may have an order or not, so explicit instructions about navigation are necessary

interventions

Plan for whether interventions are to be allowed and, if so, how they’ll work. Should they only be for insurmountable obstacles?

three main kinds

others?

key logging

eye tracking

think aloud

Or a reflection session to get verbal feedback in case think aloud would interfere with task performance

reminders

People don’t show up unless you send reminders

extra time

People show up late or get interrupted, so you have to allow extra time

follow protocol

May have an irb (Institutional Review Board) protocol but should have some protocol anyway

testing the artifact, not the user

The participant should be reminded of this because it will otherwise appear that the user is being tested

flexible when compared with experiments

The protocol for the session should be followed but can be looser than for an experiment

debriefing

analysis

morae

Why would you even care about Morae? It’s obsolete and discontinued. But … some people still use it and the workflow is quite good even if you don’t use it. So let’s explore it from the workflow standpoint.

suggested morae workflow

1. plan
2. setup
3. record, view, log
4. create project
5. view, analyze
6. present

morae includes three programs

but first …

Think about the kind of session you’ll run

paper prototype

If you’re testing a paper prototype, you probably need both cameras. You can set up Morae in paper prototype mode so that it does not record the PC screen.

moderated computer prototype

You can set up Morae in moderated usability test mode in a lab and run the test solo or with observers in the control room.

This is the most likely setup.

autopilot

Don’t discount the value of autopilot. This is meant for unmoderated sessions but you can use it to free you from having to juggle too many things to remember during the session. The only caveat is that you have to set it up properly or you will wind up abandoning it and returning to moderated mode.

process (one of three)

process (two of three)

process (three of three)

during the test

after the test

Now it is time get some evidence from your recordings and prepare it for presentation.

You’ll want to find video clips that illustrate important points.

You’ll want to log markers so you can get a graph that supports your findings.

alternatives

report writing Mensh and Kording (2017)

caveat: This is about academic writing and industrial writing is usually less strict

four principles

components of a report

process

usual report format

using published information

validity Lazar, Feng, and Hochheiser (2017)

Did we measure what we said we measured?

face validity

criterion validity

construct validity

establishing validity

using statistics

using grounded theory

using content analysis

design thinking process

1. empathize
2. define
3. ideate
4. prototype
5. test

… but should it really end there?

another round

more design!

iterative development

\usetikzlibrary{backgrounds}
\usetikzlibrary{shapes.geometric}
\tikzstyle{background rectangle}=[rounded corners,fill=yellow!05]
\definecolor{protocolor}{rgb}{0.2,0.60,0.25} 
\definecolor{designcolor}{rgb}{0.9,0.1,0.0} 
\definecolor{evalcolor}{rgb}{0.3,0.4,0.5} 
\definecolor{arrowcolor}{rgb}{0.7,0.8,0.9} 
  \begin{tikzpicture}[show background rectangle,scale=1.35] 
\node (p) at (0,2)
  [shape=ellipse,draw=none,fill=protocolor,text=white,scale=0.85]
  {\sf prototype};
\node (pa) at (-1.5,2.6)
  [shape=rectangle,draw=none,fill=protocolor!60,text=white,scale=0.5]
  {\sf lowfi, hifi};
\node (d) at (4,3)
  [shape=ellipse,draw=none,fill=designcolor,text=white,scale=0.85]
  {\sf design};
\node (da) at (4.7,4.3)
  [shape=rectangle,draw=none,fill=designcolor!60,text=white,text
  width=4cm,scale=0.5]
  {\sf contextual inquiry, personas, scenarios, sketching};
\node (e) at (2,0)
  [shape=ellipse,draw=none,fill=evalcolor,text=white,scale=0.85]
  {\sf evaluate};
\node (ea) at (4,-0.4)
  [shape=rectangle,draw=none,fill=evalcolor!60,text=white,text
  width=5cm,scale=0.5]
  {\sf heuristic evaluation, usability testing};
\draw [->,very thick,arrowcolor] (d) to [bend right=45] (p);
\draw [->,very thick,arrowcolor] (p) to [bend right=45] (e);
\draw [->,very thick,arrowcolor] (e) to [bend right=45] (d);
  \end{tikzpicture} 

product of evaluation

more design ideas

should you do whatever participants asked?

NO!

At least, not necessarily

what can you do?

Understand participants better first!

… before you try to make design choices

first thoughts may scratch the surface

ask why? three times

why keep asking why?

to get beneath the surface

evaluation is not really finished

until you understand the usability problems

when do you understand the problems?

when you’re ready to make design choices

AI in heuristic evaluation

General problem

Most AI tools have poor usability and may not get better in the short term for a few reasons, mainly that people have a poor conceptual model of how they really work

You need to work with them for a while to get a feel for how to get the most out of them

Specific problems

Mixed views

How to do it

You can upload a screenshot to several AI tools and a list of heuristic guidelines and get back a list of problems

You can, for example, use the OpenAI Vision API for this, but it is not free

My best recommendation right now is to use free tools; this is because good prompts to limited models seem in my experience to get better results than poor prompts to more sophisticated models and you need some practice to generate good prompts