It came as no surprise to me that running lean experiments was going to be a popular topic in our Slack channel, but I was shocked by just how popular.
The questions came in thick and fast but Amanda Stockwell handled them with ease. She’s an old hand at Ask the UXperts sessions and we’re lucky she keeps coming back.
Amanda is the perfect person to talk about research methods. She has three popular courses on different aspects of user research. If you haven’t already, you can check out those courses here.
If you didn’t make the session today because you didn’t know about it, make sure you join our community to get updates of upcoming sessions.
If you’re interested in seeing what we discussed, or you want to revisit your own questions, here is a full transcript of the chat.
Transcript
hawk
2017-05-25 22:00
Ok all, let’s get this show on the road.
hawk
2017-05-25 22:00
First, a huge thanks to @amandastockwell for her time today
hawk
2017-05-25 22:00
And to you guys for joining – esp those that are new
hawk
2017-05-25 22:01
Quick overview of how we’ll roll:
hawk
2017-05-25 22:01
I’ll intro Amanda, she’ll intro the topic, and then we’ll throw it open for questions
hawk
2017-05-25 22:01
If it gets busy I’ll queue them in a back channel, so you can ask at any time – you don’t have to wait for a gap
hawk
2017-05-25 22:01
I’ll mark your questions like this so you know they’ve been acknowledged
hawk
2017-05-25 22:02
And I’ll post up a full transcript of the session on http://uxmastery.com tomorrow
hawk
2017-05-25 22:02
So the formal introduction:
hawk
2017-05-25 22:03
Amanda Stockwell is President of Stockwell Strategy, a UX research practice focused on lean research methods and integrating user knowledge with business goals to create holistic product strategies for organizations of all sorts.
hawk
2017-05-25 22:03
She has focused most of the last decade focused on finding innovative ways to understand end users and embed that knowledge into overall process.
hawk
2017-05-25 22:03
She’s lead teams that provide research, design, and UX strategy services and frequently writes and speaks about her experience. She has a human factors background and an engineering degree from Tufts University.
hawk
2017-05-25 22:03
And she has a couple of great courses on LinkedIn http://lynda.com/LinkedInLearning
amandastockwell
2017-05-25 22:04
Ha that does sound formal! Thanks for great intro
hawk
2017-05-25 22:04
Is that the right link to your courses?
amandastockwell
2017-05-25 22:04
try this instead – https://www.lynda.com/Amanda-Stockwell/5287222-1.html
amandastockwell
2017-05-25 22:05
and ps shout out to @maadonna who helped with the card sort one :slightly_smiling_face:
amandastockwell
2017-05-25 22:05
anyway, on to lean experiments! so as mentioned, I’ve been in UX about a decade, primarily focused on research and strategy
frankenvision
2017-05-25 22:05
The course on UX Interviews is awesome
amandastockwell
2017-05-25 22:05
and the last few years I’ve been more in the product/business strategy space, which means I’ve been in contact more and more with the ideas of lean startup
amandastockwell
2017-05-25 22:06
thanks @frankenvision!
amandastockwell
2017-05-25 22:06
So lean startup is really just an approach to building businesses and products that is intended to reduce risk
amandastockwell
2017-05-25 22:07
because so often there are startups or parts of big businesses that think they have good ideas, spend tons of time and resources building them, and then they launch to …crickets. no one cares
amandastockwell
2017-05-25 22:08
if you’ve not read it, there’s a book called Lean Startup by Eric Ries that summarizes the practices
amandastockwell
2017-05-25 22:08
There’s a lot to it, but it’s basically the scientific method repackaged in business terms
amandastockwell
2017-05-25 22:09
and the key idea is to learn as soon as possible without investing too much time/money
amandastockwell
2017-05-25 22:10
So lean experiments are tools that people use to help them learn all kinds of things about their business, whether it’s how interested people really are in a product or feature to best marketing copy to use
alex.lee
2017-05-25 22:10
How would you compare your approach to Steve Krug’s Rocket Surgery approach? What are differences and similarities?
amandastockwell
2017-05-25 22:12
So, it’s been a while since I’ve read rocket surgery, so correct if I’m wrong, but that book is specifically about streamlining usability testing, right?
alex.lee
2017-05-25 22:13
Yes and doing minimal viable research
amandastockwell
2017-05-25 22:13
Ok so there is definitely some crossover, but with lean experiments, you’re not _just_ focused on usability or even the specifics of a product
sandyux1
2017-05-25 22:13
What online tools do you use to help with your scientific research?
amandastockwell
2017-05-25 22:13
you might be trying to test out whether you should build a feature (which is more like product strategy)
amandastockwell
2017-05-25 22:14
so Steve’s approach for usability testing and doing minimal research works really nicely with lean experiments, it’s just that the scope is often different
amandastockwell
2017-05-25 22:15
with lean you might be trying to learn anything about the business from what customer segment to focus on, how to market to them, etc.
amandastockwell
2017-05-25 22:15
make sense?
jellybean
2017-05-25 22:15
Why is it ‘lean’
amandastockwell
2017-05-25 22:16
For online tools – I use TONS depending on what I’m doing! sometimes my favorite is just a video conferencing tool so I can talk and watch people interact, but i also like unmoderated tools like trymyui or http://usertesting.com. optimalsort is the JAM for card sorts and tree test
frankenvision
2017-05-25 22:16
How do you frame a problem for lean experiments?
amandastockwell
2017-05-25 22:16
i like usabilityhub for some quick stuff
amandastockwell
2017-05-25 22:16
@sandyux1 do you have a specific kind of research you’re looking for tools for?
amandastockwell
2017-05-25 22:17
@jellybean it’s called lean because it has a lot of influence from the “lean manufacturing” space, which is a method of reducing waste in production. That was popularized at Toyota
sandyux1
2017-05-25 22:18
Thanks, I love optimalsort too!
amandastockwell
2017-05-25 22:18
@frankenvision i always start with a few things – the first is to consider the context of where you’re at in terms of product development – are you brand new, do you have an existing product that isnt’ doing well, etc
alex.lee
2017-05-25 22:19
Thanks @amandastockwell
amandastockwell
2017-05-25 22:19
There’s a thing in lean startup called GOOB, which just stands for get out of the building. Basically it means you do customer interviews and get out of your own space. The first thing you really need to do is get an understanding of the people you think you want to serve, what their needs and goals are, where they might have gaps in experience, and opportunities to serve them
amandastockwell
2017-05-25 22:20
not so very different than typical discovery/exploratory type research
hawk
2017-05-25 22:20
And how do you decide what kind of experiment to run?
alex.lee
2017-05-25 22:20
How do you determine if your lean experiment is quantitatively accurate, to avoid potential false positives (first thing that comes to mins is small sample size)?
amandastockwell
2017-05-25 22:21
the difference is that you then create a backlog of work but framed as assumptions. Not, we definitely need to build x y z but more like we have x y z assumptions to figure out
amandastockwell
2017-05-25 22:21
and then you prioritize based on what is riskiest – basically what would be most fatal if you get it wrong
amandastockwell
2017-05-25 22:21
pick one assumption at a time, and then turn it into a hypothesis, just like an old school science experiment
amandastockwell
2017-05-25 22:22
the framework i like to use for hypotheses is: If we (do, build, provide a thing), then (our subset of people), will (do something we want them to). We’ll know when (some sort of metric that proves it)
amandastockwell
2017-05-25 22:23
did that answer your question for framing?
frankenvision
2017-05-25 22:23
How do you convince stakeholders to buy into lean experiments? Is it best to start something quickly and present findings?
amandastockwell
2017-05-25 22:25
@hawk the experiment should be based on what you can do to test and prove or disprove the hypothesis you’ve picked.
amandastockwell
2017-05-25 22:26
so let’s say you’re at the beginning stages and want to prove that a particular app idea is compelling. You might do something like create fake landing pages for 2 different ideas and see which one gets the most traffic or the most page engagement
amandastockwell
2017-05-25 22:27
@alex.lee – that’s a tough one, because one of the things about lean is that you have to figure out what is the minimal way to prove or disprove your thesis, and sometimes that means you’re a little less comprehensive than you might otherwise be
amandastockwell
2017-05-25 22:28
but the thing is to set your success metric as part of the hypothesis and to also decide on failure points before you set your experiment, than try to design your experiment to answer that question as best as possible
frankenvision
2017-05-25 22:28
what do you do when lean experiments fail on tight deadlines?
alex.lee
2017-05-25 22:28
So the lean approach maybe more suited to formative research phase?
amandastockwell
2017-05-25 22:28
so again, say you do the landing page thing and want to see which page gets most traffic. You might design as part of the experiment that you won’t look at results until there have been 1,000 total responses or something
amandastockwell
2017-05-25 22:29
@frankenvision: For stakeholders, my advice for lean experiments is same as basically anything – don’t ask permission! Just do it, prove how useful and awesome it is, and then try to convince :slightly_smiling_face:
amandastockwell
2017-05-25 22:30
not as easy as it sounds, but in general, i’m a fan of finding a way to just try something and then going back with results
amandastockwell
2017-05-25 22:30
no one is ever mad when you come at them with a lot of learning
alex.lee
2017-05-25 22:30
Got it.
frankenvision
2017-05-25 22:30
Thanks
amandastockwell
2017-05-25 22:30
and since lean experiments are, by design, not meant to take up lots of time or resources, it can be easy to sneak in. harder when changing an existing product or something though
amandastockwell
2017-05-25 22:30
sure thing!
amandastockwell
2017-05-25 22:31
also, I’m so glad you asked about “failed” experiments – here is the thing. The only way to fail at experimenting is TO NOT DO IT.
amandastockwell
2017-05-25 22:31
because by doing a little test and finding out something doesn’t work, that means you shouldn’t put in any more effort!
amandastockwell
2017-05-25 22:31
that’s good! that saves you time!
sandyux1
2017-05-25 22:31
:+1:
diferifar
2017-05-25 22:32
How do you prioritize what experiment will you be running first?
amandastockwell
2017-05-25 22:32
now, sometimes the findings go against something that a hippo (highest paid person) or seagull (swoop and poop) wants and that can be tough
kstillwell
2017-05-25 22:32
Is the statistical significance of a lean experiment necessary for it to carry weight? If you can identify trends early on in an experiment, why not just adjust and move on? Do you have any experience with doing just that?
amandastockwell
2017-05-25 22:32
but that’s not different in lean experiments vs any other sort of political debate
amandastockwell
2017-05-25 22:33
re: formative phase – nope, I think lean experiments can be run all throughout the product development cycle, you just shift what you’re experimenting on
amandastockwell
2017-05-25 22:33
maybe once you have something fairly well built out, you shift your experimenting on best ways to market or play with different price points or new features
amandastockwell
2017-05-25 22:33
I think its awesome for formative but not only for it
amandastockwell
2017-05-25 22:34
@diferifar for prioritizing experiments, I base that on how risky it is to be wrong about it
amandastockwell
2017-05-25 22:34
so for instance, assumptions I have about what sort of thing to build or who to build it for are riskier than the particulars of a specific interface
frankenvision
2017-05-25 22:35
How many lean experiments do you run at one time?
amandastockwell
2017-05-25 22:35
that sometimes has to be a judgement call and I don’t have a super scientific way, but I recommend making the priority discussion something that’s done among a larger team
pammyele2017
2017-05-25 22:35
Can you carry out an experiment indoors for starters and if so, what do you recommend for an indoor lab setup?
diferifar
2017-05-25 22:35
Got it, ty
amandastockwell
2017-05-25 22:36
cool! @kstillwell you can call trends early, but truthfully you’re not usually ever going for statistical significance in a lean experiment – you’re going for the minimum amount to prove or disprove your point
amandastockwell
2017-05-25 22:36
so it can be real risky to cut something short when you see a trend because you can get false results
frankenvision
2017-05-25 22:37
What methods do you use to analyze data for lean experiments?
amandastockwell
2017-05-25 22:37
that being said, sometimes it’s super obvious what direction something is moving and if so, then what I’ll usually do is move on to a next hypothesis that builds off the last one. so if i was doing landing page testing of 3 concepts and concept 1 seemed by far the best, i might stop and then design an experiment just to see how well concept 1 does on its own
amandastockwell
2017-05-25 22:38
the good thing about lean experiments is that they are designed to be inputs for more experiments, so you always get more opportunity to explore and answer questions
amandastockwell
2017-05-25 22:38
re: how many experiments to run. start with 1!
amandastockwell
2017-05-25 22:39
and get it done quickly, then move on to next
amandastockwell
2017-05-25 22:39
it is possible to multiple at once on different things (kind of like multivariate testing) but it gets hard to manage and focus
amandastockwell
2017-05-25 22:39
@pammyele2017 can you clarify your question?
amandastockwell
2017-05-25 22:40
do you mean in house, like without external participants?
amandastockwell
2017-05-25 22:41
re: data analyzation methods – that totally depends on the type of experiment you run, but it tends to skew more quantitative than the pure ux work I do. I’m often looking at things like site analytics and click through rate, number of downloads, that kind of thing.
amandastockwell
2017-05-25 22:41
and if you set a clear enough metric in your hypothesis, analysis should be really easy! you either hit your goal or no
amandastockwell
2017-05-25 22:42
like, in an example from an article I wrote, let’s say you’re testing app concepts and your hypothesis is as following: If we build an app that automatically generates 5 recipe ideas per week,
Then busy parents,
Will be interested in downloading this application.
We’ll know this is true when we present them with a variety of food-related apps and they choose the recipe generation app at least 15 percentage points more often, for example, than any other choice.
frankenvision
2017-05-25 22:42
ok thanks
swan5280
2017-05-25 22:42
Hi Amanda, do you have any advice for working with important stakeholders who view taking to customers as an enemy of innovation?
amandastockwell
2017-05-25 22:43
you figure out a way to test which one they’ll pick and then the analysis is simple – the recipe generation app is either 15 percentage points more or it’s not
amandastockwell
2017-05-25 22:43
sure thing!
pammyele2017
2017-05-25 22:43
@amandastockwell – I mean internal within the company. Or if they are external testers, done with remote tools. Although I am part of a large company, my team itself is limited in how much effort we can make to GOOB :slightly_smiling_face:
amandastockwell
2017-05-25 22:44
hi @swan5280! i assume you mean “talking” to users? If so…my advice is run away :slightly_smiling_face:
swan5280
2017-05-25 22:44
Ha yes. Talking
amandastockwell
2017-05-25 22:45
just kidding! i know some people have the “if you asked people what they want they don’t know” attitude and that’s true, but people are invaluable sources of information. deeply understanding the people you hope to serve and their needs is the best way to uncover gaps in their experience (and those are your opportunities!)
amandastockwell
2017-05-25 22:45
that doesn’t tell you HOW to solve the problem but it tells you what problems are most compelling
amandastockwell
2017-05-25 22:46
for real though, convincing people to let you talk to people you’re designing for could and should be it’s very own ask the experts session :slightly_smiling_face:
amandastockwell
2017-05-25 22:46
i think there might even have been a related one
swan5280
2017-05-25 22:46
:+1:
pammyele2017
2017-05-25 22:47
I totally agree with Amanda about people being invaluable. I launched a product last August and depite all the testing we did before release. The best feedback came after release and people started working and using the platform.
amandastockwell
2017-05-25 22:47
@pammyele2017 so it depends who you are building stuff for – but if you’re trying to build stuff for people who don’t work at your company, i say get out when you can. remote is a totally great option though
jellybean
2017-05-25 22:47
If you wanted to test something like a landing page or a homepage are your hypothesis related to specific tasks you want users to do, or overall impressions, or specific features, or all of the above?
amandastockwell
2017-05-25 22:48
and a lot of lean experiments don’t require you to talk to people – they’ll be measuring how many people downloaded something or clicked on a link (which REALLY needs to be with external people lest your results be TOTALLY biased or you’re working on something for internal people)
hawk
2017-05-25 22:49
We have 10 minutes left so if you’re sitting on a question, get it in now.
pammyele2017
2017-05-25 22:49
Thanks @amandastockwell
amandastockwell
2017-05-25 22:50
@jellybean well you’d set your hypothesis first, then design the experiment to test that. So your hypothesis might be about a specific task or a specific feature or how many people click on something or any number of other things. the landing page is just a tool to get at that data
amandastockwell
2017-05-25 22:51
so i wouldn’t start with building a landing page, i’d start with the question you want to answer and the assumption you have, then work backward
amandastockwell
2017-05-25 22:52
or if you already have a landing page, you could start with an assumption you have (like, we assume that people’s first impression is that our site is trustworthy)
amandastockwell
2017-05-25 22:52
then work to test that
jellybean
2017-05-25 22:52
Ahhh… Thank you. :slightly_smiling_face:
amandastockwell
2017-05-25 22:52
sure thing @pammyele2017!
amandastockwell
2017-05-25 22:52
sure! anyone else have clarifying questions or anything else?
amandastockwell
2017-05-25 22:53
we covered a lot in a short amount of time :slightly_smiling_face:
hawk
2017-05-25 22:53
We did!
sandyux1
2017-05-25 22:53
:clap:
flaxenink
2017-05-25 22:53
Awesome!! Quick question is there a book about metrics for designer?
flaxenink
2017-05-25 22:53
:tired_face:
diferifar
2017-05-25 22:53
Is there any specific measurement you check besides those already into your experiments?
amandastockwell
2017-05-25 22:54
ooh good question about books – i know tons of good ones but not about metrics – let me think!
amandastockwell
2017-05-25 22:55
and @diferifar yes, but totally depends what project/product i’m working on!
amandastockwell
2017-05-25 22:56
soo for instance, right now i have a project that is a pretty straightforward a/b test of a few different landing pages for an engagement campaign. every day we run a report that looks at bounce rate, time on page, and clicks on certain elements. the hypothesis is about which design will work best for a specific action but we’re tracking all the rest
flaxenink
2017-05-25 22:56
Kinda like metrics or analytics for a designer coming into into that world… :clap:
diferifar
2017-05-25 22:56
Project transversality measurements, maybe? Probably working on different projects, but same industry?
amandastockwell
2017-05-25 22:57
hmm so I know of a good search analytics book – Search Analytics for Your Site – Rosenfeld Media
amandastockwell
2017-05-25 22:57
not sure of a similar one for general analytics
amandastockwell
2017-05-25 22:58
i’ll think on it!
hawk
2017-05-25 22:58
And I think that’s a wrap!
hawk
2017-05-25 22:58
Huge thanks again for your time today Amanda. You rocked it.
kstillwell
2017-05-25 22:58
thank you!
hawk
2017-05-25 22:58
Thanks everyone for joining us
amandastockwell
2017-05-25 22:58
thanks everyone! and feel free to reach out if you think of more questions!
alex.lee
2017-05-25 22:58
Thanks so much @amandastockwell