Magitest: A Better Approach to Mobile User Testing

Magitest: A Better Approach to Mobile User Testing

An illustration of a magician's hat, a wand and a mobile phone
Summary:

Have you ever wished you had an easier way to user test your mobile app (or the mobile experience of visiting your website)?

A new iPhone app, Magitest, promises to change that by turning your phone into a portable mobile usability lab. Matt puts the app through its paces.

This is a review of Magitest, an iPhone app for conducting mobile user testing. The author was provided with a review copy of the app.

Summary

  • Name of app: Magitest
  • Platform(s): iPhone only; iPad, Android coming soon.
  • Description: Allows you to perform mobile user testing of native apps and websites. Captures the participant’s voice and facial expressions during the session.
  • Price: $24.99 (in-app purchase; free trial is time-limited to 60 seconds of recording time)
  • Rating: 9/10

Here’s a video of me using Magitest with the browser pointed at uxmastery.com:

Full Review

A few years ago, I had an idea for an app.

I know, I know. Everyone has an idea for the next killer app, right? At the time, the app store was still in its relative infancy, and early stories of successful appreneurs were beginning to surface.

I’d been using Silverback, the simple but reliable user testing software from Clearleft (still my user testing tool of choice; see other similar tools). Silverback is basically just screen-capture software, but it cleverly makes use of the computer’s built-in microphone to record audio of the session, and the built-in camera to record video of the participant’s face, which is then overlaid onto the video in the corner of the screen. With Silverback, I’m able to conduct user testing sessions anywhere—at a client site, in a coffee shop, on a train—as long as I have my laptop with me. No need to book an expensive lab with a one-way mirror. Silverback has not only been instrumental for me in doing more user testing, but also in selling the idea to clients: “It’s so easy for us to run a test—why wouldn’t we?”

The Magitest logo
Magitest: making mobile user testing look like magic

My idea was to create a “Silverback for iPhone”: an app that used the front-facing camera and built-in mic to turn the iPhone into a portable mobile user testing lab. It would be insanely useful for me, so I figured that other UXers would want it too.

Once I began to research the idea, I realised the technical challenges were entirely non-trivial, and I didn’t take it any further.

The Magitest team, on the other hand, have built just that app.

How I Used To Test Mobile Apps

Why is it so useful to have an app that lets you conduct user testing on your phone? Well, if you’ve ever conducted a mobile user test before, you’ll know that doing so is not straightforward. There are three basic options:

  1. Use a tiny camera mounted to the phone with some kind of purpose-built rig.

    What I dislike about this approach is that any rig, no matter how lightweight or ingeniously engineered to hold the camera, is intrusive. Participants may find the extra weight of the rig disconcerting, aside from the visual distraction of a strange robotic arm protruding from the phone; this hurts our attempts to simulate a real-world scenario.

  2. Use a video camera positioned to record over the participant’s shoulder.

    This is an approach I’ve had more success with—I’ll attach an inexpensive flip-cam attached to a monitor arm with rubber bands. While less intrusive, you need to instruct the participant to hold the phone at a specific height to ensure that the video camera captures the screen. There can also be issues with low audio as they are talking away from the camera.

  3. Capture the user’s session on another device.

    It’s possible to use AirPlay to send a user’s session to another computer or AppleTV. I’ve also experimented with using an app called LiveView, which does the reverse and allows a user to interact with a desktop app (which can then be recorded on that computer). This approach can work well, but it doesn’t record any audio, is reliant on having a strong network connection, and isn’t an option for native apps.

Enter Magitest

Magitest is the app that I wanted to create. I’m not suggesting any appropriation of ideas or anything—the concept is kind of obvious, and execution is everything. But it’s exactly what I envisioned: Magitest is Silverback for the iPhone. If you’re a Silverback user, you’ll understand that this is a compliment: Magitest does one thing—record session data on your phone—and does it well.

A screenshot of the Magitest start screen
The Magitest start screen: a couple of options, and away you go.

The Pros

  • Support for native apps.

    This is a huge selling point for Magitest. There are several apps out there that open a browser window and allow you to record the user session, but none of them work for native apps.

  • No equipment required.

    As I mentioned above, being able to conduct guerilla user testing on a phone without a video camera, camera stand, phone mount, cable, or anything else is huge.

  • No network required.

    Because Magitest performs all of its capture and post-processing of video on the device, there’s no worrying about whether your session will be lost due to network dropouts. It also opens up the number of places you can conduct user testing “in the wild.”

  • Easy to use.

    Magitest is as easy to use as you might expect for an app whose premise is to encourage the user testing of mobile apps. There’s no hypocrisy here—no user manual required.

  • Price.

    At $24.99, sure—Magitest is more expensive than the majority of games, productivity and novelty apps that dominate the Best Seller lists of the app store. However, when you consider the audience (people who perform user testing of mobille apps for a job) it’s a bargain (e.g. Silverback for Mac is $70).

  • Aesthetic.

    Functionality aside, Magitest looks polished. It has a pretty icon, plays well on the magician’s aesthetic, and utilises typography that reinforces this branding message. All of this adds up to a more enjoyable experience.

That said, there are a couple of misgivings that I had about the app.

The Cons

  • Speed

    The biggest hurdle I found with Magitest was that speed was an issue. It may be unfair to pin this on the app, as if you think about what is happening in parallel (screen render, screen capture, audio capture and video capture) then it is a big ask of your phone. I didn’t have an iPhone 5 handy to test Magitest out on, but I imagine that Magitest would be a lot zippier when testing both native apps and websites on the 4S or iPhone 5. If upgrading your phone is not an option, be prepared for your users to experience a bit of lag time while interacting with your app or website.

  • Heat

    Somewhat related, I did notice my phone got a little warm after using Magitest for only a couple of minutes. This wasn’t necessarily a big hurdle, but it may be smart to use a plastic case that makes this heat increase less obvious to the user if you think it might become a distraction.

  • Setup

    I didn’t get a chance to try Magitest out with a native app, mostly because the process requires that you export your app from Xcode. If you’re working closely with your development team, this is probably not a hurdle at all. However, it does mean you can’t user test production versions of your app, or do any competitive research by user testing other similar apps in your space. It may be a little unfair to list this as any kind of failing on the app’s behalf—it’s simply a necessity given the restrictions that Apple place on iOS apps.

Summary

Magitest is a great concept that has been executed well and I’m certain will change the face of mobile user testing. It is very demanding of resources, but I imagine this would be the case for any app that attempts what Magitest does. So while it was a passable experience on my iPhone 4, if I was going to be doing a ton of user testing for a mobile device, I would definitely upgrade my phone so that the performance was improved. There are also versions for iPad and Android in the pipeline, and I’m sure that this app will be very popular among UX designers.

Overall, Magitest definitely gets the thumbs up, and I intend to use it to test all of my mobile and web projects moving forward. I give the app a magical 9 out of 10.

Magitest - Magitest
Download Magitest from the iTunes app store.

Written by
Matthew Magain
Join the discussion

9 comments
  • Haha! I love that to end the Magitest test session you need to shake the phone. Hope you didn’t get a headache at the end there, Matt!

  • Thanks for the review Matt.

    We’d like to reinforce your comment about Magitest running a bit slowly on the iPhone 4. As you mention, it uses a lot of system features (screen capture, camera and microphone in addition to running the site/app that you’re testing) and therefore a lot of system resources.

    Apple introduced a much more powerful processor with the iPhone 4S and the performance boost is significant. While Magitest will generally work (albeit a little slowly) on the iPhone 4, we recommend using a 4S or newer.

    We also recommend using iOS 6 or later as the “Guided Access” feature and “Do Not Disturb” can help prevent your tests being inadvertently ended by the user pressing the home button or by an incoming call. There’s more info about this on our support page and on our blog.

    Happy testing.

  • I love this idea – I haven’t tried the app yet but definitely will. My only reservation with this approach – like using airplay & broadcasting to a larger screen to allow observation is that you don’t capture (on video) how a user interacts or tries to interact with the UI – you don’t get this information if you can’t see their fingers & the touch gestures they use or the screen elements they (try to) interact with before whatever takes them to the next state – the next state is the only thing that will be captured (unless there’s a facilitator who manages to capture all of this and pass it on). In my experience this is really important for touch screens as designers & developers tend to massively overestimate the number & complexity of touch gestures used by most people. That said – I don’t have a solution other than a rig which as you say is intrusive & cumbersome. I look forward to trying the app in any case! :)

    • Thanks for commenting Bex! You might have noticed in the video that Magitest does display a small purple circle that indicates where the user touches, which goes some way to addressing your concerns. That said, I would still consider a facilitator to be necessary to guide the user if they were to get stuck—to make them feel at ease, and to make note of UI hurdles, as you say. This app makes running the test a lot easier, but I don’t see it making the facilitator redundant at all.

  • This was awesome, great to know we have “an App for that” :)

    One question, is the recording a simple video with tap spots, how do you present this, is there a sample video that I can access.

    Also, are there any plans to produce some clickstream reporting with analytics, something like that will be too cool.

Follow @uxmastery

Instagram has returned empty data. Please authorize your Instagram account in the plugin settings .

Instagram

Instagram has returned empty data. Please authorize your Instagram account in the plugin settings .