Transcript: Ask the UXperts—Usability Testing with Gerry Gaffney

Transcript: Ask the UXperts—Usability Testing with Gerry Gaffney

Gerry Gaffney
Summary:

“Ask the UXperts” is a casual chat-based session featuring a different expert each time. It is One Hour. One Expert. All your Questions Answered.

In today’s session we featured the wonderful Gerry Gaffney and we discussed Usability Testing. Here’s a transcript of the session.

Today I hosted an interesting addition to our series of Ask the UXperts sessions. If you haven’t heard of these sessions before,  Ask the UXperts can be summarised as One Expert. One Hour. All your questions answered. And that was certainly the case today.

Our expert was the wonderful Gerry Gaffney, and we discussed Usability Testing

Gerry is the company founder and lead consultant at infodesign.com.au and has lectured undergraduate and postgraduate students in various universities on usability and user experience design. He is passionate about the need to improve the quality of the relationship between humans and the products we use and runs the User Experience Podcast uxpod.com. Gerry was the ultimate expert today, not only answering questions, but asking them in return. The chat was informal and relaxed, but a lot was covered.

Timezones are the bane of my life and they mean that someone always misses out, so if that was you this time (or if you’d just like to revisit what we discussed or check out some of the resources mentioned), you’ll find the full transcript below. If you’d like further information about future sessions, make sure you join our community where I’ll keep you in the loop.

For those of you that would like to see what we talked about today, here is a full transcript for your perusal. Grab a coffee, it’s a long one…

HAWK
Gerry Gaffney: Hey!
 
A tweet will go out shortly and people will start joining us
HAWK
Justin: If you want Gerry’s undivided attention on Usability Testing, now would be a good time to take advantage of that ;)
 
I’m about to undertake my first usability test
Gerry G.
do you do usability testing?
 
awesome!
Justin T.
I’m 6 months into a gig – part time
Gerry G.
are you excited or nervous about doing it?
Justin T.
I’ve created a prototype and through dropbox im going to host it.
 
Not particularly nervous. More about the knowledge requirement
Gerry G.
is it one-on-one, or remote testing?
Justin T.
Remote
 
remote through a closed FB group and in person
 
Trying out the gurilla testing as the client doesn’t have the budget for professional testing
Gerry G.
For the remote will it be moderated (i.e. will you be online as people complete tasks) or unmoderated?
 
Budget is often the issue!
Justin T.
Still to be worked out
Gerry G.
I think though that the single most powerful way of selling usability testing is having someone in the business observe a test in person
Justin T.
There will have to be some sort of script that the users will just read vs a chat system of sorts using free tools
Gerry G.
Whatever works is my rule of thumb.
Justin T.
yeah – This is one of the first cases of testing being involved in a ‘design’ project
 
I’m all about getting things done and learning from it
Gerry G.
Make sure you do a pilot because you’ll likely find something you’ve overlooked.
 
at least, i always overlook something!
Justin T.
yeah – I’m going to test and run a complete session in house first. Polish then set loose
Gerry G.
Hi Keith T. Justin T and I are just chatting about remote testing.
Gerry G.
“Polish then set loose”. Love it!
Keith T.
Thanks
Justin T.
Are you aware of any software to ‘heatmap’ or track through clicks of an Axure prototye?
Gerry G.
Gosh, I dont’ know Justin T.
Justin T.
reading your review materials on your website Gerry
Keith T.
LOL – Just trying to finish off a last minute call
Gerry G.
Have you had a look at the book “Beyond the Usability Lab”?
Keith T.
I’ll be with you as soon as I can
Justin T.
seems like it must have been someone else asking the question before me
 
not yet
Justin T.
I usually ask the question (of the internet) then build it
Gerry G.
In that book the authors talk about specific tools and compare them.
Justin T.
I made the first ‘paper prototype’ as a clickable pdf, then was told it must be clickable – so I learnt Axure and made one
HAWK
Justin Thor Hoyer: Ask the UXperts next week is with a guy that knows all about heat mapping, so he may know the answer to that one
Justin T.
cool
 
ahh, thanks
Gerry G.
Axure is great. There are so many tools out there now. My most recent prototype was using HTML/CSS and Twitters Bootstrap framework
 
But becuase I’m not really technical it took longer than a “real programmer” would do it in.
 
Thanks Hawk, that’s cool, we’re just chatting really… Happy for you to make it more formal at your leisure!
Justin T.
drag and drop is definatly the future. Especially as we want to have the speed of paper, but the interactivity of the web
 
Is there an agenda for this session?
Gerry G.
I still use paper for the early stuff. It’s just so quick and valuable to get feedback early
HAWK
We don’t need to make it official at all, if you guys are happy. All I usually do is ask the UXpert to give us a brief bio and an intro to the subject. Then I throw it open to the group to ask questions. If things get busy, I queue them for Gerry.
Justin T.
cool
Gerry G.

Sure! I’ve been running usability tests for more than 15 years now.

I do UX work in commercial and government, and occasionally lecture (less so now).

I have a company called Information & Design and I run the User Experience podcast (uxpod.com)

HAWK
And a brief intro to UT would be useful for the transcript
Gerry G.
Usability testing is probably the think that taught me more than anything else I’ve done!
 
Usability testing is a technique for ensuring that the intended users of a system can carry
out the intended tasks efficiently, effectively and satisfactorily.
 
that’s a very old definition I wrote maybe 10 years ago.
 
There are lots and lots of variants on the technique.
HAWK
Great definition though. KISS.
Gerry G.
And you can be as infomral or formal as you like.
HAWK
Do you have a preference, or does it depend on the audience?
Gerry G.
There are hardly any really serious mistakes you can make, in my opinion
 
Depends on the circustances, for me
 
I like to go for the lowest cost option
 
that will do the job effectively
 
and the lowest resolution appropriate for the tasks. So that might be paper, might be a full-fledged prototype, might be somewhere in between
Justin T.
what ‘package’ do you give the client as a summary of results of the testing? Documentation, Video, or just a one pager
HAWK
For those of you that just joined, this is an informal session and you are welcome to jump in with questions at any time. You don’t have to wait for a gap – I’ll queue questions if necessary
Gerry G.
Depends on the client. In a very formal enviornment,a very formal report listing issues and recommendations and describing the methodlogy
 
For a client I know well, something much more punchy
 
And if I’m in an Agile development process, migth be a one-pager and briefing to the team
 
My preference is short & sweet
 
rather than a big document that sends everyone to sleep.
Justin T.
Minimum viable in – most important change out
Gerry G.
exactly
 
I think it can be a mistake to either over-engineer or under-engineer your report.
Justin T.
How long did it take you to specialise in your career from ‘All UX’ to usability testing?
Gerry G.
Justin Thor Hoyer, actually it was kind of the other way around
 
Usability testing was one of the first things I did.
 
It was a real eye-opener for me, as well.
Justin T.
Gerry Gaffney:
 
nice ux pod site
Gerry G.
Justin Thor Hoyer, thanks!
Justin T.
how so?
Gerry G.
Well, before that I’d only informally watched people using technology, so I knew there were problems but I didn’t really understand the extent of how bad things were for most people
 
Ordinary people were unable to use technology that was supposedly designed to help them.
 
And people at the time I think were more likely to blame themselves
Justin T.
What’s the most interesting project you’ve worked on recently?
Gerry G.
We’d hear a lot of “I’m no good with computers”
Justin T.
It’s all just a mental model and relatable context isn’t it.
Gerry G.
Justin Thor Hoyer, most interesting project is I just finished the design of a new Jury Management System
 
That involved staff screens on desktop, forms, leters and mobile-first for jurors
Justin T.
thinking about taking on the election process next? we need it
Gerry G.
Justin Thor Hoyer: Whitney Quesenbery in US has started a “center for civic code” and she’s an expert on election UX
 
It’s her thing.
 
Might be worth looking her up: wqusability
Justin T.
someone should own it
Gerry G.
Yeah :>)
Justin T.
it’s an interesting thing. What problem is worth solving vs what problem can you solve today?
HAWK
Gerry Gaffney: So I have a question. You said that there are ‘hardly any’ serious mistakes that a person could make. Can you tell us what one of those mistakes would be?
Gerry G.
Justin Thor Hoyer: that’s right
 
HAWK: OK, the number 1 is… (drum roll)
 
testing with the wrong participants.
 
I’ve had quite a few projects where people claimed to have tested the product but it turned out to be colleagues or people who were experts or otherwise inappropriate.
 
The key mantra is “representative users”
Kory
have you ever done ux testing where you used virtual reality goggles?
Gerry G.
Um, no. I’ve done a little bit of testing with eye-trackers, but not enought to really talk about.
 
Kory, have you got such a proejct? sounds like fun!
Amando
Any rule of thumb for the quantity of representative users?
Gerry G.
@AM the quantity of users is a real political question in many areas. here’s my take:
Kory
have a project coming up related to the auto industry & we need to simulate a driving experience
Gerry G.
If you’re testing a working prototype, around 6 people is *generally* good enough to identify the marjotity of issues
 
However, if you have wildly differeint user groups (say an airline booking site that has to support novices and expert travel agents) you may need to do more.
 
Kory: I’ve use Monash Uni’s driving simulator at MUARC (Monash Uni Accident Research Centre).
 
that might be worth looking into…
 
But even testing with 1 user is good
 
And for very early design there’s a fine line between testing thats summative and testing that’s formative
 
So summative or evaluative testing will be about: Is this product working as we expect?
 
and formative will be about: What can I learn from my users to help me design this product?
 
I find in reality that slipping between the two “modes” can be useful
 
although there are times you want do do one or the other.
Kory
Good research article on how many users to test with: http://www.simplifyinginterfaces.com/wp-content…
Gerry G.
Kory: I know there’s lots of research in driving and UX, both from a safety perspective and from in-car technology perspective
Amando
At what stage do you allow rep users to test? I assume you use a prototype tool? What tool do you use for wire framing prior to design prototyping?
Baldur K.
Any pointers in user testing using paper mock-ups?
Gerry G.
@Aman I get representative users from day 1, if at all possible
Kory
We need to do this testing in-house. Looking into driving simulators.
Gerry G.
Amando: @Baldur for prototypes, I use different tools
 
Axure, Balsamiq
 
Sometimes just clickable PDFs made in Powerpoint
 
I love Omingraffle
 
ALso HTML/CSS
 
To answer @Baldur K, testing with paper…
Todd
proto.io is a great tool to make high interaction fidelity prototypes
Justin T.
I know it’s a print tool, but to me Indesign is much better/faster than omnigraffle.. Thoughts?
Gerry G.
I think paper is a great way to go. A lot of people say otherwise, but my experience is that it’s quick, it’s cheap and it’s really effective.
Todd
full iOS app simulation. no coding
 
i use it to test with users and use reflector to mirror on my screen so they can use the app on their own phone, naturally
Gerry G.
Paper allows you to do cool things like mockups of physical devices. I’ve given people “phones” that are just foam board with screens made of paper covered in acetate and they buy right into the experience and give really good feedback because it’s obviously exploratory and not a “final” version.
 
I haven’t used proto.io
 
We’re kind of spoiled for choice.
Todd
totally.
 
more my week it seems
Gerry G.
I find I’m learning a new tool at least every 6 months.
 
which on the upside is great
 
and one the downside is a pain :>)
 
I really enjoyed making an IVR mockup using Twitter’s bootstrap framework
 
But to me, the real value in usability testing is not the tools…
 
It’s observing what I call “real people” interacting with our stuff
Todd
the more motion design becomes important, the more that prototyping interactive animation becomes important to get feedback on how the app feels.
Gerry G.
It’s so powerful to see the assumptions you’ve made being validated or disproved
 
Todd, absolutely, and you can’t do those sorts of interactions with paper
Baldur K.
That’s great idea ( the foam phone ) … definitely gives them more “natural” sense of using the product
Gerry G.
Dan Saffer spoke about those sorts if interactive designs, and how you can’t do them on paper, inhttp://uxpod.com/microinteractions-an-interview…
 
Baldur Kristjánsson: K, yeah the foam phone is awesome
 
Same for tablet, I’ve done an android app using that tecnique and it got really good feedback
 
One thing about physical mockups….
 
is that I was really really surprised, on several projects, about how much they impress management. It’s like, suddenly they realise, “Oh, so *that’s* what these guys are doing! Now I get it!”
 
So great for getting buy-in
 
Kory, have you chosen a driving simulator yet?
Gerry G.
Another technique I really like is “RITE” –
 
rapid iterative testing & evaluation. Whereby you do a prototype, test it, refine it immediately and test next day with next user/s
Justin T.
Gerry G.
There’s an article on RITE here: http://www.usabilityprofessionals.org/uxmagazin…
 
People want to talk about a particular aspect or problem? (otherwise i can just to stream-of-consciousness…)
 
OK, so I’ll go on to some general thoughts and people can jump in with questions or whatever…
HAWK
I have another question… what are the pros and cons of remote vs one on one testing?
Justin T.
Process > Case studies > Trends… whatever works for you
Gerry G.
I think having a formal plan is good.
 
HAWK: I think you can’t beat one-on-one for depth
 
But remote of course allows you to a) get to people who you couldn’t otherwise (I did some testing with mining engineers and ued this)
 
and b) you can get a lot more partcipants if you go for unmoderated
 
example I recnetly tested a tree structure using Optimal Workshop’s “Treejack”
 
really easy to set up, and you can send out invitations to dozens or hundreds of people and voila, you’ve tested your categorisation scheme
 
(You still have to fix it of course!)
 
I think a key problem with remote is if people need to do setup
 
For example, if people have to install somethign or enable somethign, you can almost guarantee you will have some problems.
 
So there’s definitely a place for remote – see the book “Beyond the Usability Lab” for great info
Justin T.
Could you not have a setup (front page) then click to proceed, and a ‘Thanks for participating – now click here to claim your prize’ type scenario?
 
not so in depth, but doable
Gerry G.
Justin Thor Hoyer: indeed you can
 
Although it’s hard to know what’s *actually* going on.
 
So it’s great for tests where you want to know where people went, or what task completion rates were
 
but you miss out on the one-on-one interactions
Justin T.
yeah – heat map and tracking links…
 
so do both (time and funds allowing)
Gerry G.
http://uxpod.com/beyond-the-usability-lab-an-in… is worth a read/listen on remote testing
 
Justin Thor Hoyer:
 
On any given project, I might mix both methods, if I have both the need and the resources.
Yasmine
What online tools/software do you use to help with remote usability testing?
Gerry G.
I guess i tend to feel that oen-on-one is so much more valuable that it tends to be my fvaourtie
Justin T.
yeah – I know (I’m excitable)
Gerry G.
Yasmine: I’ve been farily simple in my approach – Treejack that I already mentioned, but “GoToMeeting” or any tool that allows screen-share is great if you want to to remote *moderated* testing
Justin T.
Thanks heaps Gerry – gotta run. I’ve followed you on twitter and I’ll check out teh podcasts.
Gerry G.
That way you have someone in a remote location but you can observe their interaction in real time and talk to them about what they’re doing.
 
Cheers @Justin T
Yasmine
Gerry – Thanks for the response, came in late, and can’t see the conversation that had happened before I came in.
Gerry G.
Ah, I know that @HAWK is putting up a transcript later
HAWK
Yasmine: I’ll post a full transcript up on uxmastery later today so you can see what you missed
Yasmine
I appreciate it!
Gerry G.
any other specific questions?
 
Want to talk about planning, scripts, questionnaires, tasks, etc…
 
disasters…
George
What do you think about a short “screening” pre chat with candidates to check they’re right for the full testing session?
 
Is that always necessary / useful / essential / difficult for user recruiters?
Gerry G.
George, hi George, screening is important. Depends on the cirucmstance, but I firstly use an agency to recruit whenever possible
 
Not alwasy necessary to use recruiters
 
But it saves an awful lot of time and effort, and you don’t end up gettign people’s cousins!
George
What have you found to be the strongest alternatives to using recruiters? (Apologies if covering previous ground!)
Gerry G.
George: well, connections can be good, but if you’re not careful you can end up with non-representtative users – and that’s a real no-no in my opinion
 
Well, non-repsresentative can sometimes be OK, but if you test with experts you may well get a false sense of security
 
you think your product is OK but in the real world it bombs
George
Yes, that makes sense.
Gerry G.
But you can’t always use recruiters
Todd
Gerry, …, I need to head out, but thanks so much for your insights. btw, here’s the link tohttp://www.proto.io to check out for creating such high interaction fidelity prototypes.
Gerry G.
for example, I needed some fairly senior judges recently, and an agency wasn’t going to get them
George
And I would imagine that successful relationships with recruiters depend on good briefs to them. Any tips on that?
Gerry G.
Todd: thanks!
 
George: for the brief…
 
1. Write down the key criteria (might be age, might be employment area, usage of a particular service or whatever), make a table of those things, make sure your client is across it and then send it off to the recruiter.
George
Cool, thanks.
Gerry G.
Make sure you have a phone conversation with the recruiter, and also review and script they are using to recruit
 
and then i normally have a questionnaire during testing that validtes whetehr we got the right people
Amando
If developing a native mobile app, do you prototype both android and windows for testing?
Gerry G.
Generally I’ve found that if you recruit 10 people, 9 will be really useful and one will be weird but intereseting!
George
Thanks, Gerry.
Gerry G.
Amando: I wouldn’t (generally) see the need to test on multiple platforms to a *great* extent
 
For example…
Harlequin
Funny, I’m in the middle of finding a new opportunity…trying to make yourself that one interesting person =)
Gerry G.
Our jury management service is HTML5/CSS and nominally runs on any mobiel device. I used an Android (rooted) to record so that was my first choice
 
Harlequin :>)
Matthew M.
I love creating highlights reels from usability testing sessions (for internal use of course). I always make sure I include one of the “weird” ones ;-)
Gerry G.
But I also tested on iPad a few times.
 
as a kind of backup.
 
Matthew Magain, love highlight videos!
 
But I try to make sure that I can put my hand on my heart and say “here is representative footage”
 
Sometimes the weird ones are just too weird
 
Incidentally Tedesco & Tranquada have a new book called “The Moderator’s Survival Kit” which talks about a lot of the things that can happen!
Matthew M.
Indeed! With great power comes great responsibility. Responsible curation can be challenging!
Gerry G.
They’ll be on uxpod.com in a few weeks.
Harlequin
Q: If you’re coming into a company that hasn’t had any designer as its lead UX/UI person what’s a good way to sell yourself with your “Here’s how I can help you” quote, unquote “speech”…
Gerry G.
Harlequin: I think the key thing is to do something useful.
 
That might be help review a screen, or something fairly mundane, but demonstrative value is more powerful than any spiel
Harlequin
Noting, this is also a product I would have not seen until said interview…it’s not public yet…
Gerry G.
A client talked about this recently on: http://uxpod.com/ux-a-client-perspective/
Harlequin
I do have examples of previous work for sure…a portfolio and stories behind them are important…
 
I’ll check it out, thx
Gerry G.
Harlequin: and another powerful thing is offer to run a test… say with only 1 or 2 people, and have key staff observe
 
there’s nothing as pwerful in my opinion as having people observe real humans interacting with their product
 
That’s the real fun in usability testing
Harlequin
As in a “what I’ve seen so far” with their product? A quick, non-rude feedback session?
Gerry G.
It can be tedious setting up and planning and doing the number crunching afterwards…
 
Harlequin: as long as they respect you, then yes, but there’s also a danger if they see it being in any way an attack
 
Not that you’d attack them of course!
 
But it can put people on the defensive.
Harlequin
Yeah, keeping it on the down-low for sure :)
Gerry G.
What about “this is a great product, let’s test it with a few users”?
 
Do it really really cheaply so nobody can object on budget
danielle.v
Watching users struggle does make a big impact. I’m finding it hard to improve even small things such as horrendous error messages or instructions. The IT or project team feel like they make sense and don’t understand why it would be confusing to Joe Bloggs.
Harlequin
I like that…
Gerry G.
i mean, you can run a test with a couple of hundred dollars TOPS.
 
unless you need to recruit nuclear engineers
 
I think we’re about at wrap-up time – any last-minute comments?
Harlequin
Adding you to Twitter, realized I saw this in a retweet..lol
HAWK
That was a really interesting chat, thanks so much for your time Gerry.
Matthew M.
Thanks Gerry, thanks all!
Sarah Hawk
Written by
Sarah Hawk
Join the discussion

Follow @uxmastery

Invalid username or token.

Instagram

Invalid username or token.