117 Words

Unmoderated User Testing

A time-efficient and cost-effective method to test end-users.

Let us paint you a picture. The month is July. You have just been given a project on a super tight holiday deadline. You have five weeks to get the product shipped and out to consumers. So you rush headlong to complete the project, forging the testing phase.

Then the holidays hit. And … what comes next is one of the largest failures known to man. Your company dedicates a landfill out in Alamogordo, New Mexico for the purpose of burying millions of returned and unsold products. Does this story sound familiar? It should because it is 100% true. This is what happened with the infamous "E.T. the Extra-Terrestrial" video game for the Atari 2600 — those who guessed correctly, mark yourself 10 points.

The point of the story: if you do not test for your audience — or test at all — you are doomed to failure. Without an extra set of eyes to look at your product, you are just designing for yourself. Your bias will hinder you from seeing all the problems that could have be easily fixed. Those problems can be the difference between commercial success and catastrophic failure.

User testing is important in making sure that end-users who visit your site are able to navigate and use all the functions therein properly.

The three primary types of user testing:

  • In-Person

  • Remote

  • Asynchronous

Each of these methods have their pros and their cons. But we're thinking ahead of the curve, setting our sights on using more unmoderated user testing. And here's why.


First, let's pit these abled warriors of user testing against one another, and let’s see just how they stack up against one another.

In-Person (Moderated) User Testing

This is far and away the most widely known and accepted method of testing, user or otherwise. A proctor sits in on the testing session and asks questions while the participant carries out a task. This allows for a series of instantaneous give-and-take feedback, as well as qualitative information transactions. With this type of test, you can quickly think through your usability flaws and correct them as you see fit, changing them for the next round of participants.

Amid the positive aspects of moderated testing, there lurk a few obstacles to this model of testing.

  • Expense. This is a serious Achilles' heel, both in time and expense. Travel expenses, lab rentals, test equipment and time away from your business are just a few of the costs. In a small startup environment, these resources are essential to your everyday operations and cannot be bandied about.

  • Location. More often than not, these tests are carried out in "laboratory settings" or other types of settings that can be regarded as "unnatural" for participants. Taking away a participants' sense of comfort will hinder test results as they feel pressured to do things a certain way. This can cause far more errors than just taking their time as they would in a more comfortable setting.

  • Bias. Though this can be limited with a properly run moderated test, we are still human and mistakes, miscues and unconscious body language behavioral cues can transpire. An unintended consequence of this: the moderator could potentially carry over biased, leading information and non-verbal cues to the participant, which will slant results.

Though most of these drawbacks for in-person testing are found with formal moderated testing, there have been methods developed to lessen some of these disadvantages. Guerrilla testing is just one type of in-person test. This method is pure, unadulterated do-it-yourself ethos in action. The designer simply asks people, more often than not coworkers, to do a quick test. This beats the need to set up a facility to test. Saving a bit of expense.

Another advancement for user testing is dealing with the lack of "comfortable" lab settings. This hindrance was overcome by moving the testing methods to a remote moderated system.

Remote User Testing

Remote moderated testing allows the participant to sit at their own computer in their own home and take your tests, you proctor the test via software that allows you and the participant to view one another.

This alleviates any need for rental labs or travel expenses. However, you still need to allocate the proper software and set aside specific time slots. But we don't always have spare time to waste for this, especially as deadlines loom.

We have to start planning for more cost-efficient, time-effective testing. We need to take all the cons of guerrilla and moderated testing, and turn them into opportunities. At the same time, we need to take the pros of each, mashing them into a single type of user test.

Asynchronous (Unmoderated) User Testing

This term/idea/concept is so new to the world of user testing that the word "unmoderated" gets that vicious red-squiggly underline when it's input into most typing prompts.

With unmoderated user testing, willing participants check out your web design from a remote location so you can attain quantitative data — by tracking assigned tasks.

The remote locations that participants choose to test at can be as comfortable as their living rooms. By giving the participant the relaxation of performing the test in real life locations, it relieves any tension they might have in an artificial moderated testing environment. Aside from non-artificial locations, users can achieve context through their ability to use phones and tablets during these tests. In other words, they can test anywhere and on any device.

Unmoderated user testing is not proctored. Using software that tracks and records every move, allows remote testers to make notations and observations that an in-person moderator would typically make. And the participants don't have to be in the same room as a moderator. They can take the test any time of the day, whether they be on the far side of the world or just across the street.

And for the financially minded, this is a real boon because there's relatively low overhead since you don't have to set up a lab or conduct the test yourself. That means, your return-on-investment looks much larger in the greater picture of your business.

For comparison, it was noted that the total moderated usability test cost, of even a low-end test, would be estimated at about $10,500 in 1994 — adjusted for 2012 is approximately $16,400.

That’s one hulking chunk o' chicken feed. With an unmoderated user test, you're eliminating the need to rent or reserve a lab. You're saving time since your presence isn't required, so you can get other tasks done in your day. By not travelling, as in the case of some moderated user testing, some travel by airplane is necessary — that’s a lot of time and money alone. With travel there is the propensity of damaged or lost equipment.

To make the long story short, unmoderated testing not only saves money and time, but you get more results and are able to draw much actionable data as well.

An All-In-One Testing Tool

At ZURB, we like to stay ahead of the curve, which is why we've built Solidify for everyone to take advantage of unmoderated user testing (sorry for the shameless plug). The app offers a "responsive testing experience" (you can test on multiple devices whether it be on phone, tablet, laptop, desktop, etc.).

Solidify can implement testing in a variety of ways, whether it be in person, remotely or at the participants own time. It can adapt to any situation that you'd like to utilize the app. In the words of Grand Moff Tarkin, "Now, witness the power of this fully-armed and operational unmoderated user testing software!" OK, he may or may not have used those exact words.

Though each of these types of tests have their individual set of positive and negative characteristics, there is one product from all the tests that we need: RESULTS. With these results we can begin to factor how to proceed with our designs. How we determine how to proceed is with the use of confidence intervals.

Confidence Intervals

Once all the data has been recorded and charted, we implement the use of confidence intervals because it's impossible to sample the entire world of possible users. This algorithm estimates just how likely our garnered sample size results will apply out in the wild.

Thanks to these intervals, it does not matter if you have a sample size of 20,000 or 20, all data can be made actionable. In doing so, we can make sure we are confident enough to go forward with certain design decisions for more favorable usability.

Unmoderated User Testing and You

Step By Step: Conduct Your Own Unmoderated User Test

To begin your unmoderated user testing, you should first develop a succinct, easy-to-follow questions and tasks that you would like your participants to answer and complete.

The need for clear assigned tasks is paramount. In this testing experience your participants are left to themselves. If they don't understand a question or task they cannot ask for help or explanation. How can you expect to receive decent results or feedback from a participant who is lost and confused? Keeping your participant free of uncertainty is vital. If we learned anything from Castlevania II, it's that non-linear and cryptic instructions will definitely hinder your participants’ experience to the point of unconstrained frustration.

Let’s Talk it Out

Since you, the would-be proctor of testing, are not present with the unmoderated testing model, an important element for your test is to encourage your participants to think out loud. By recording the participant and having them verbalize their actions as they proceed with their tasks or questions, you can begin understand user behavior from an unbiased perspective. These opinions help you sculpt a better, more usable design from more than just the quantitative test feedback.

Though, not all unmoderated user testing supports visual or audio recordings of participants (yet), you can still utilize text inputs throughout the test in order to gain those qualitative results.

Getting Your "Students"

That's right, we have to have participants in order to carry this testing thing out. There are several sites or resources you can check out to enroll such participants, but don't trouble yourself with scouring the vast web for the best prices or best candidates. Luckily, we, ZURBians, got a little something-something up our sleeves. We have our own network of user testers called Enroll. The network is chock-full unbiased and demographically diverse participants to use at your whim. No need to search high and low for willing candidates. One theory to keep in mind, with how many participants you choose with your test and how they work into your budget, is that of Jakob Nielsen's five user theory.

The Five User Theory

Nielsen explains how when testing usability, as soon as your first participant completes their test, the feedback contains approximately a third of everything you need to know about the usability of your design.

After each round of testing, the results contain less and less new information regarding your designs. There are just more and more overlap of the first noted problems. By the fifth test completion, you should have just about 85% of your usability problems documented. In order to understand 100% of your usability, Nielsen says that at least 15 participants are more than likely needed. However, that many testers for one test is a waste of your precious resources (aka time and money).

The idea with the testing is to document opportunities for success with better design rather than just harping on each and every weakness. After your five participants finish their tests, that's when you should revamp your design and then test again. When you test your new design, that’s when the remaining 15% should be caught be the next round of five testings.

Rinse and repeat as needed — redesign and retest as needed.

Nielsen's five user theory presents a strong case for those working on a budget, but still want the appropriate feedback on their site iterations. The theory can even be used for those who have budgets ranging from modest to large who would like to cut back on resources being eaten up by unnecessarily large usability tests.

Time for the "Final Examination"

Now that you have squared away some participants, it's time to execute the test itself.

There are two different ways to connect with your participants. We can have the participants either connect to a site that carries out the test, or have them download software which implements the test.

Once you decide which avenue you’d prefer, it's a simple matter of uploading what you want the participants to test and then inputting the tasks and questions. Then simply direct your participants to the link or software for them to use. From this point, you can kick back and enjoy a nice cold beverage of your choice while the software records the tests and you begin to see the results and feedback trickle in.

Let the Computers Work their Magic

During your relaxation period, the software will be recording the workflow of the participants. You can see how they clicked and where they clicked through the design to fulfill their given tasks and questions. This is important to see how easy it is for first time users to navigate through your design and see if the usability is as functional as you had originally desired.

Additionally, a meaningful batch of metrics can be recorded with the software. These quantitative metrics include:

  • Task completion rates;

  • Time on task;

  • Time on pages;

  • And web analytic data (web analytic data includes the participants choice of: browser, operating system and screen resolution)

Basically, each of these aspects are just a more in-depth analysis of the workflow. By looking at the time spent to complete tasks, regardless of completed correctly or not, you can determine how effective your design is for the participants.

Because of the flexibility for the tester, this testing model is quickly gaining a following with those designers who are staying up-to-date with the most advanced, progressive-thinking means of maintaining their prominence on the Web.

Here's to Championing a New Era of User Testing

Unmoderated user testing is paving a new path for less expensive and less time-consuming testing methods, all the while providing just as reliable data and results as the more conventional methods have provided.

Log in with your ZURB ID

Log in with Google

Don't have a ZURB ID? Sign up

Forgot password?

Sign up for a ZURB ID

Sign up with Google

Already have a ZURB ID? Log in