Today I hosted an interesting addition to our series of Ask the UXperts sessions. If you haven’t heard of these sessions before, Ask the UXperts can be summarised as One Expert. One Hour. All your questions answered. And that was certainly the case today.
Our expert was the wonderful Gerry Gaffney, and we discussed Usability Testing.
Gerry is the company founder and lead consultant at infodesign.com.au and has lectured undergraduate and postgraduate students in various universities on usability and user experience design. He is passionate about the need to improve the quality of the relationship between humans and the products we use and runs the User Experience Podcast uxpod.com. Gerry was the ultimate expert today, not only answering questions, but asking them in return. The chat was informal and relaxed, but a lot was covered.
Timezones are the bane of my life and they mean that someone always misses out, so if that was you this time (or if you’d just like to revisit what we discussed or check out some of the resources mentioned), you’ll find the full transcript below. If you’d like further information about future sessions, make sure you join our community where I’ll keep you in the loop.
For those of you that would like to see what we talked about today, here is a full transcript for your perusal. Grab a coffee, it’s a long one…
Gerry Gaffney: Hey! | |
A tweet will go out shortly and people will start joining us | |
Justin: If you want Gerry’s undivided attention on Usability Testing, now would be a good time to take advantage of that ;) | |
I’m about to undertake my first usability test | |
do you do usability testing? | |
awesome! | |
I’m 6 months into a gig – part time | |
are you excited or nervous about doing it? | |
I’ve created a prototype and through dropbox im going to host it. | |
Not particularly nervous. More about the knowledge requirement | |
is it one-on-one, or remote testing? | |
Remote | |
remote through a closed FB group and in person | |
Trying out the gurilla testing as the client doesn’t have the budget for professional testing | |
For the remote will it be moderated (i.e. will you be online as people complete tasks) or unmoderated? | |
Budget is often the issue! | |
Still to be worked out | |
I think though that the single most powerful way of selling usability testing is having someone in the business observe a test in person | |
There will have to be some sort of script that the users will just read vs a chat system of sorts using free tools | |
Whatever works is my rule of thumb. | |
yeah – This is one of the first cases of testing being involved in a ‘design’ project | |
I’m all about getting things done and learning from it | |
Make sure you do a pilot because you’ll likely find something you’ve overlooked. | |
at least, i always overlook something! | |
yeah – I’m going to test and run a complete session in house first. Polish then set loose | |
Hi Keith T. Justin T and I are just chatting about remote testing. | |
“Polish then set loose”. Love it! | |
Thanks | |
Are you aware of any software to ‘heatmap’ or track through clicks of an Axure prototye? | |
Gosh, I dont’ know Justin T. | |
reading your review materials on your website Gerry | |
LOL – Just trying to finish off a last minute call | |
Have you had a look at the book “Beyond the Usability Lab”? | |
I’ll be with you as soon as I can | |
seems like it must have been someone else asking the question before me | |
not yet | |
I usually ask the question (of the internet) then build it | |
In that book the authors talk about specific tools and compare them. | |
I made the first ‘paper prototype’ as a clickable pdf, then was told it must be clickable – so I learnt Axure and made one | |
Justin Thor Hoyer: Ask the UXperts next week is with a guy that knows all about heat mapping, so he may know the answer to that one | |
cool | |
ahh, thanks | |
Axure is great. There are so many tools out there now. My most recent prototype was using HTML/CSS and Twitters Bootstrap framework | |
But becuase I’m not really technical it took longer than a “real programmer” would do it in. | |
Thanks Hawk, that’s cool, we’re just chatting really… Happy for you to make it more formal at your leisure! | |
drag and drop is definatly the future. Especially as we want to have the speed of paper, but the interactivity of the web | |
Is there an agenda for this session? | |
I still use paper for the early stuff. It’s just so quick and valuable to get feedback early | |
We don’t need to make it official at all, if you guys are happy. All I usually do is ask the UXpert to give us a brief bio and an intro to the subject. Then I throw it open to the group to ask questions. If things get busy, I queue them for Gerry. | |
cool | |
Sure! I’ve been running usability tests for more than 15 years now. I do UX work in commercial and government, and occasionally lecture (less so now). I have a company called Information & Design and I run the User Experience podcast (uxpod.com) | |
And a brief intro to UT would be useful for the transcript | |
Usability testing is probably the think that taught me more than anything else I’ve done! | |
Usability testing is a technique for ensuring that the intended users of a system can carry out the intended tasks efficiently, effectively and satisfactorily. | |
that’s a very old definition I wrote maybe 10 years ago. | |
There are lots and lots of variants on the technique. | |
Great definition though. KISS. | |
And you can be as infomral or formal as you like. | |
Do you have a preference, or does it depend on the audience? | |
There are hardly any really serious mistakes you can make, in my opinion | |
Depends on the circustances, for me | |
I like to go for the lowest cost option | |
that will do the job effectively | |
and the lowest resolution appropriate for the tasks. So that might be paper, might be a full-fledged prototype, might be somewhere in between | |
what ‘package’ do you give the client as a summary of results of the testing? Documentation, Video, or just a one pager | |
For those of you that just joined, this is an informal session and you are welcome to jump in with questions at any time. You don’t have to wait for a gap – I’ll queue questions if necessary | |
Depends on the client. In a very formal enviornment,a very formal report listing issues and recommendations and describing the methodlogy | |
For a client I know well, something much more punchy | |
And if I’m in an Agile development process, migth be a one-pager and briefing to the team | |
My preference is short & sweet | |
rather than a big document that sends everyone to sleep. | |
Minimum viable in – most important change out | |
exactly | |
I think it can be a mistake to either over-engineer or under-engineer your report. | |
How long did it take you to specialise in your career from ‘All UX’ to usability testing? | |
Justin Thor Hoyer, actually it was kind of the other way around | |
Usability testing was one of the first things I did. | |
It was a real eye-opener for me, as well. | |
Gerry Gaffney: | |
nice ux pod site | |
Justin Thor Hoyer, thanks! | |
how so? | |
Well, before that I’d only informally watched people using technology, so I knew there were problems but I didn’t really understand the extent of how bad things were for most people | |
Ordinary people were unable to use technology that was supposedly designed to help them. | |
And people at the time I think were more likely to blame themselves | |
What’s the most interesting project you’ve worked on recently? | |
We’d hear a lot of “I’m no good with computers” | |
It’s all just a mental model and relatable context isn’t it. | |
Justin Thor Hoyer, most interesting project is I just finished the design of a new Jury Management System | |
That involved staff screens on desktop, forms, leters and mobile-first for jurors | |
thinking about taking on the election process next? we need it | |
Justin Thor Hoyer: Whitney Quesenbery in US has started a “center for civic code” and she’s an expert on election UX | |
It’s her thing. | |
Might be worth looking her up: wqusability | |
someone should own it | |
Yeah :>) | |
it’s an interesting thing. What problem is worth solving vs what problem can you solve today? | |
Gerry Gaffney: So I have a question. You said that there are ‘hardly any’ serious mistakes that a person could make. Can you tell us what one of those mistakes would be? | |
Justin Thor Hoyer: that’s right | |
HAWK: OK, the number 1 is… (drum roll) | |
testing with the wrong participants. | |
I’ve had quite a few projects where people claimed to have tested the product but it turned out to be colleagues or people who were experts or otherwise inappropriate. | |
The key mantra is “representative users” | |
have you ever done ux testing where you used virtual reality goggles? | |
Um, no. I’ve done a little bit of testing with eye-trackers, but not enought to really talk about. | |
Kory, have you got such a proejct? sounds like fun! | |
Any rule of thumb for the quantity of representative users? | |
@AM the quantity of users is a real political question in many areas. here’s my take: | |
have a project coming up related to the auto industry & we need to simulate a driving experience | |
If you’re testing a working prototype, around 6 people is *generally* good enough to identify the marjotity of issues | |
However, if you have wildly differeint user groups (say an airline booking site that has to support novices and expert travel agents) you may need to do more. | |
Kory: I’ve use Monash Uni’s driving simulator at MUARC (Monash Uni Accident Research Centre). | |
that might be worth looking into… | |
But even testing with 1 user is good | |
And for very early design there’s a fine line between testing thats summative and testing that’s formative | |
So summative or evaluative testing will be about: Is this product working as we expect? | |
and formative will be about: What can I learn from my users to help me design this product? | |
I find in reality that slipping between the two “modes” can be useful | |
although there are times you want do do one or the other. | |
Good research article on how many users to test with: http://www.simplifyinginterfaces.com/wp-content… | |
Kory: I know there’s lots of research in driving and UX, both from a safety perspective and from in-car technology perspective | |
At what stage do you allow rep users to test? I assume you use a prototype tool? What tool do you use for wire framing prior to design prototyping? | |
Any pointers in user testing using paper mock-ups? | |
@Aman I get representative users from day 1, if at all possible | |
We need to do this testing in-house. Looking into driving simulators. | |
Amando: @Baldur for prototypes, I use different tools | |
Axure, Balsamiq | |
Sometimes just clickable PDFs made in Powerpoint | |
I love Omingraffle | |
ALso HTML/CSS | |
To answer @Baldur K, testing with paper… | |
proto.io is a great tool to make high interaction fidelity prototypes | |
I know it’s a print tool, but to me Indesign is much better/faster than omnigraffle.. Thoughts? | |
I think paper is a great way to go. A lot of people say otherwise, but my experience is that it’s quick, it’s cheap and it’s really effective. | |
full iOS app simulation. no coding | |
i use it to test with users and use reflector to mirror on my screen so they can use the app on their own phone, naturally | |
Paper allows you to do cool things like mockups of physical devices. I’ve given people “phones” that are just foam board with screens made of paper covered in acetate and they buy right into the experience and give really good feedback because it’s obviously exploratory and not a “final” version. | |
I haven’t used proto.io | |
We’re kind of spoiled for choice. | |
totally. | |
more my week it seems | |
I find I’m learning a new tool at least every 6 months. | |
which on the upside is great | |
and one the downside is a pain :>) | |
I really enjoyed making an IVR mockup using Twitter’s bootstrap framework | |
But to me, the real value in usability testing is not the tools… | |
It’s observing what I call “real people” interacting with our stuff | |
the more motion design becomes important, the more that prototyping interactive animation becomes important to get feedback on how the app feels. | |
It’s so powerful to see the assumptions you’ve made being validated or disproved | |
Todd, absolutely, and you can’t do those sorts of interactions with paper | |
That’s great idea ( the foam phone ) … definitely gives them more “natural” sense of using the product | |
Dan Saffer spoke about those sorts if interactive designs, and how you can’t do them on paper, inhttp://uxpod.com/microinteractions-an-interview… | |
Baldur Kristjánsson: K, yeah the foam phone is awesome | |
Same for tablet, I’ve done an android app using that tecnique and it got really good feedback | |
One thing about physical mockups…. | |
is that I was really really surprised, on several projects, about how much they impress management. It’s like, suddenly they realise, “Oh, so *that’s* what these guys are doing! Now I get it!” | |
So great for getting buy-in | |
Kory, have you chosen a driving simulator yet? | |
Another technique I really like is “RITE” – | |
rapid iterative testing & evaluation. Whereby you do a prototype, test it, refine it immediately and test next day with next user/s | |
There’s an article on RITE here: http://www.usabilityprofessionals.org/uxmagazin… | |
People want to talk about a particular aspect or problem? (otherwise i can just to stream-of-consciousness…) | |
OK, so I’ll go on to some general thoughts and people can jump in with questions or whatever… | |
I have another question… what are the pros and cons of remote vs one on one testing? | |
Process > Case studies > Trends… whatever works for you | |
I think having a formal plan is good. | |
HAWK: I think you can’t beat one-on-one for depth | |
But remote of course allows you to a) get to people who you couldn’t otherwise (I did some testing with mining engineers and ued this) | |
and b) you can get a lot more partcipants if you go for unmoderated | |
example I recnetly tested a tree structure using Optimal Workshop’s “Treejack” | |
really easy to set up, and you can send out invitations to dozens or hundreds of people and voila, you’ve tested your categorisation scheme | |
(You still have to fix it of course!) | |
I think a key problem with remote is if people need to do setup | |
For example, if people have to install somethign or enable somethign, you can almost guarantee you will have some problems. | |
So there’s definitely a place for remote – see the book “Beyond the Usability Lab” for great info | |
Could you not have a setup (front page) then click to proceed, and a ‘Thanks for participating – now click here to claim your prize’ type scenario? | |
not so in depth, but doable | |
Justin Thor Hoyer: indeed you can | |
Although it’s hard to know what’s *actually* going on. | |
So it’s great for tests where you want to know where people went, or what task completion rates were | |
but you miss out on the one-on-one interactions | |
yeah – heat map and tracking links… | |
so do both (time and funds allowing) | |
http://uxpod.com/beyond-the-usability-lab-an-in… is worth a read/listen on remote testing | |
Justin Thor Hoyer: | |
On any given project, I might mix both methods, if I have both the need and the resources. | |
What online tools/software do you use to help with remote usability testing? | |
I guess i tend to feel that oen-on-one is so much more valuable that it tends to be my fvaourtie | |
yeah – I know (I’m excitable) | |
Yasmine: I’ve been farily simple in my approach – Treejack that I already mentioned, but “GoToMeeting” or any tool that allows screen-share is great if you want to to remote *moderated* testing | |
Thanks heaps Gerry – gotta run. I’ve followed you on twitter and I’ll check out teh podcasts. | |
That way you have someone in a remote location but you can observe their interaction in real time and talk to them about what they’re doing. | |
Cheers @Justin T | |
Gerry – Thanks for the response, came in late, and can’t see the conversation that had happened before I came in. | |
Ah, I know that @HAWK is putting up a transcript later | |
Yasmine: I’ll post a full transcript up on uxmastery later today so you can see what you missed | |
I appreciate it! | |
any other specific questions? | |
Want to talk about planning, scripts, questionnaires, tasks, etc… | |
disasters… | |
What do you think about a short “screening” pre chat with candidates to check they’re right for the full testing session? | |
Is that always necessary / useful / essential / difficult for user recruiters? | |
George, hi George, screening is important. Depends on the cirucmstance, but I firstly use an agency to recruit whenever possible | |
Not alwasy necessary to use recruiters | |
But it saves an awful lot of time and effort, and you don’t end up gettign people’s cousins! | |
What have you found to be the strongest alternatives to using recruiters? (Apologies if covering previous ground!) | |
George: well, connections can be good, but if you’re not careful you can end up with non-representtative users – and that’s a real no-no in my opinion | |
Well, non-repsresentative can sometimes be OK, but if you test with experts you may well get a false sense of security | |
you think your product is OK but in the real world it bombs | |
Yes, that makes sense. | |
But you can’t always use recruiters | |
Gerry, …, I need to head out, but thanks so much for your insights. btw, here’s the link tohttp://www.proto.io to check out for creating such high interaction fidelity prototypes. | |
for example, I needed some fairly senior judges recently, and an agency wasn’t going to get them | |
And I would imagine that successful relationships with recruiters depend on good briefs to them. Any tips on that? | |
Todd: thanks! | |
George: for the brief… | |
1. Write down the key criteria (might be age, might be employment area, usage of a particular service or whatever), make a table of those things, make sure your client is across it and then send it off to the recruiter. | |
Cool, thanks. | |
Make sure you have a phone conversation with the recruiter, and also review and script they are using to recruit | |
and then i normally have a questionnaire during testing that validtes whetehr we got the right people | |
If developing a native mobile app, do you prototype both android and windows for testing? | |
Generally I’ve found that if you recruit 10 people, 9 will be really useful and one will be weird but intereseting! | |
Thanks, Gerry. | |
Amando: I wouldn’t (generally) see the need to test on multiple platforms to a *great* extent | |
For example… | |
Funny, I’m in the middle of finding a new opportunity…trying to make yourself that one interesting person =) | |
Our jury management service is HTML5/CSS and nominally runs on any mobiel device. I used an Android (rooted) to record so that was my first choice | |
Harlequin :>) | |
I love creating highlights reels from usability testing sessions (for internal use of course). I always make sure I include one of the “weird” ones ;-) | |
But I also tested on iPad a few times. | |
as a kind of backup. | |
Matthew Magain, love highlight videos! | |
But I try to make sure that I can put my hand on my heart and say “here is representative footage” | |
Sometimes the weird ones are just too weird | |
Incidentally Tedesco & Tranquada have a new book called “The Moderator’s Survival Kit” which talks about a lot of the things that can happen! | |
Indeed! With great power comes great responsibility. Responsible curation can be challenging! | |
They’ll be on uxpod.com in a few weeks. | |
Q: If you’re coming into a company that hasn’t had any designer as its lead UX/UI person what’s a good way to sell yourself with your “Here’s how I can help you” quote, unquote “speech”… | |
Harlequin: I think the key thing is to do something useful. | |
That might be help review a screen, or something fairly mundane, but demonstrative value is more powerful than any spiel | |
Noting, this is also a product I would have not seen until said interview…it’s not public yet… | |
A client talked about this recently on: http://uxpod.com/ux-a-client-perspective/ | |
I do have examples of previous work for sure…a portfolio and stories behind them are important… | |
I’ll check it out, thx | |
Harlequin: and another powerful thing is offer to run a test… say with only 1 or 2 people, and have key staff observe | |
there’s nothing as pwerful in my opinion as having people observe real humans interacting with their product | |
That’s the real fun in usability testing | |
As in a “what I’ve seen so far” with their product? A quick, non-rude feedback session? | |
It can be tedious setting up and planning and doing the number crunching afterwards… | |
Harlequin: as long as they respect you, then yes, but there’s also a danger if they see it being in any way an attack | |
Not that you’d attack them of course! | |
But it can put people on the defensive. | |
Yeah, keeping it on the down-low for sure :) | |
What about “this is a great product, let’s test it with a few users”? | |
Do it really really cheaply so nobody can object on budget | |
Watching users struggle does make a big impact. I’m finding it hard to improve even small things such as horrendous error messages or instructions. The IT or project team feel like they make sense and don’t understand why it would be confusing to Joe Bloggs. | |
I like that… | |
i mean, you can run a test with a couple of hundred dollars TOPS. | |
unless you need to recruit nuclear engineers | |
I think we’re about at wrap-up time – any last-minute comments? | |
Adding you to Twitter, realized I saw this in a retweet..lol | |
That was a really interesting chat, thanks so much for your time Gerry. | |
Thanks Gerry, thanks all! |
