Product designers are rushing to adapt to our increasingly mobile world. With advancements in tools that allow us to record behavior and collect data quickly, they are turning to unmoderated user testing so they can test without facilitator bias.
Depending on your source, there are dozens of ways to user test. Most of them overlap, but we break it down into: moderated user testing methods, whether in-person or remote, or unmoderated user testing methods, which are asynchronous.
While each type of user testing has its advantages and disadvantages, the benefits of unmoderated tests outweigh those of a moderated one.
Recruiting Users is Costly, But There's a Cheaper Way
We won't say the economy sucks, it's a little too cliche. But it certainly has been ... on a downturn. Large corporations are slashing out what they believe to be frivolous expenses, sometimes at the risk of hurting the quality of their products.
This ends up costing companies a lot more money in the long run. Maybe its a decrease in sales. Or maybe it's money and time spent on fixing usability issues. The last thing you want to do, however, is skip out on testing your new application or website on potential users. Had Josh Levy and Ross Cohen done user testing ahead of time, they wouldn't have blown $550,000 on a product no one wanted.
Luckily, unmoderated user testing is cheaper — a lot cheaper — than an in-person or remote test. Not only does it save time that can be spent on designing, you also don't need to drop tons of money for each participant.
In-person testing usually comes with incentives. Let's face it, it's why the people are there. Maybe it's a $50 Amazon Gift Card, a gift certificate to a free meal at Chili's or cold hard cash. Either way, unmoderated user testing is more flexible, and you can get a lot more participants for a lot less money and still have great results.
Cost doesn't only build up from tangible expenses either. As the old saying goes, time is money. If your team is dedicating a couple hours sitting down or hopping on the phone with remote users you also need to calculate the time spent, which could be used for other projects.
Consider this. You have an employee making $60,000/year will theoretically cost $231 for every 8 hours they spend facilitating user tests in addition to the incentive costs. Not to mention, the money that employee could have earned for the business during that time.
Your Customers Are Global, Don't Limit Yourself Locally
At ZURB, we like to user test with an unbiased and demographically diverse group of individuals, which is why we have recruited our network of testers through Enroll. We also want to test with people, who are potential customers of our products.
In-person testing limits us to local participants, but the internet is global. Even mom-and-pop shops are making international sales through their websites. Success Magazine reported that local California business Couch Guitar Straps quickly became a global company when they unexpectedly received their first international order from Tokyo. That is just one of hundreds of examples of local, sometimes small-town, companies skyrocketing in a global market thanks to the web. So why would we limit our participants to "that nice boy down the street who likes computers" or the girl you found on Craigslist when conducting user tests?
Remote user testing also doesn't fare too well here either. This is simply a hassle, you need to coordinate meeting with time zones. There could be speaking barriers, not to mention the phone bill if you have to call them!
Unmoderated testing lets you conveniently get a sample audience from wherever you would like. Or just throw your test in the wild with an email campaign or on social media as a broader approach for collecting data.
Testing on Any Device, Anywhere Gets Results from the Right Perspective
You should be testing as many of devices as you can. If we haven't made it clear on our blog yet, mobile is here to stay. Mobile retail shopping is through the roof, social media usage and the now ubiquitous "there's an app for that" jokes. Limiting your tests to a desktop or a controlled setting where you expect users complete and undivided attention doesn't deliver accurate results, not anymore.
In an unmoderated test, users may take your test while sitting on a train, in a doctor's office or on their lunch hour. Or they can be user testing on the toilet, admit it, we all take a peek at our phones in the restroom. Having potential customers test your product in their natural environment may pick up issues with your products usability that wouldn't be clear in a controlled or laboratory setting.
Laboratory user testing is artificial. While you can control variables testers, you're fully aware that they are being testers and have the need to perform in a manner they believe to be "correct," lessening the realism of your results and the circumstances that your product may be used under.
Confidence Baring Results without Facilitator Bias
This should be obvious, but you are user testing for the results! Unmoderated User Testing removed any fascinator bias or non-verbal queues that may affect the participants' actions and thoughts.
According to Frost & Sullivan, a global growth partnership company:
The moderator can introduce significant biases into the [user test], should she/he fail to ask certain questions or delve deeply into specific areas ... skilled moderators are difficult to find, and when their services are available, they are often expensive.
Designers, marketers or anyone else on your staff may not be adequately trained to conduct a proper unbiased in-person or remote user test. This isn't to say that all moderated user testing is flawed or useless, but there's plenty of room for additional variables aside from the ones you're trying to keep constant.
To support Frost & Sullivan's claim is the Observer-Expectancy Effect, which states researchers have an inherent cognitive bias that causes them to unconsciously influence participants of experiments. We must remember that even if all the facilitators questions are the same, there still remains a bias. It can creep in either through the interpretation of responses or vocal inflection, which may change a participant's behavior during a test.
On the other hand, there is the Hawthorne Effect, which brings into question how a tester may react in conditions where they believe they're being watched. Unmoderated user testers may either be taking the test anonymously or by not giving their information until the very end. This allows them to act more naturally.
While forgetting to humanize our data could be a disaster, we also want all of our participants to be on a level playing field. An unmoderated test researcher won't get tired or forget to jot something down. Best of all, there will still be a playback of the test filled with comments that aide in individualizing results.
Really ... It's All About Confidence
What defines a successful user tester? Confidence.
Regardless of your preferred method of user testing, does the data you collect make you confident that your product is ready to deploy or give you enough information to know changes you need to make your product awesome?
This is when you can chose math or a hunch. Or a combination of both. At ZURB, we highly regard Jakob Nielsen's Five User Theory.
According to Nielsen's equation, after testing 15 users you can identify 100% of usability issues. But with only a single tester you can identity one-third of your products usability issues. This is because when you introduce more testers the data is sure to overlap. Testing with five users will allow you to discover most usability issues, iterate and test again. Remember, iterations can sometimes cause more usability issues.
Without specifically mentioning them, Nielson's theory uses the idea of Confidence Intervals.
To put it simple, especially for those with a less-than-excited aptitude for math, confidence intervals allows you to infer usability issues from the general population using a small sample size. In other words, it will let you know if your user testing data is statistically significant. And depending on the percentage of confidence you would like, will let you know if you are ready for deployment or need to take another swing at things.
When it comes to user testing, it's always a balancing act. However, advances in asynchronous testing with tools, such as Solidify among others, and the growth of mobile connectivity, we need to see unmoderated user testing as a necessary future for getting results that bring our designs and products unquestionable confidence.