Design Great Products FasterPoor Questions Lead to Bad Results
Ask the wrong question and your results will be flawed. Whether you're asking an online survey question or conducting a face to face interview the question you ask can make or break your entire experiment or interview.
Recently, I had the idea to run a poll for the greatest Super Nintendo games of all time using a Verify Memory test. It seemed like a great fit, the actual test asked people for five games they remembered. The report grouped responses into the top five related phrases. I was surprised at the results.
More than half of the people that took the test did not follow the instructions.
The plan was to simply instruct users to enter their five favorite Super Nintendo games into the form, rather than their memories of the screenshot.
Seems pretty straight forward right? Memory tests in Verify require an image, so a picture of a Super Nintendo seemed appropriate. I didn't want to skew the results by using a picture of an actual game cartridge. So after reading the instructions and clicking the button to start the test, people saw this for five seconds.
After which they were presented with this form, which asked them to deposit the names of their most fondly remembered games.
I published the test and patiently waited to bath in a sea of 64-bit nostalgia. I was not naive, I expected a few people to skip or misinterpret the instructions, and thus was very surprised that the second most popular phrase was video game, indicating that many people simply put what they remembered of the image, and not their favorites games.
Why is this? To put it generally, the test was poorly designed.
To be more specific:
- We asked a question about software (games), and then showed a picture of hardware (Super Nintendo)
- The survey did not follow the established pattern for a memory test, of collecting memories from an image
- The instructions were not attached to the form
- An image of a Super Nintendo was disimilar enough from the instructions to cause people to question what they were doing
- Images are more powerful then words, people followed the instructions implied by the image, rather than the explicit written instructions
So what?
Our takeaway from this fun little experiment is to think long and hard when phrasing a question. Whether it's Verify or a Q&A interview your question is what will make or break the results. When dealing with Verify tests make sure to:
- Have just one introduction/instructions screen
- Make your instructions a maximum few lines long
- Make the question/instructions as clear and concise as possible. You grandmother should understand them.
- Shoot the preview of your test to a few colleagues to see if they can understand and take your test without your help
Greg (ZURB) says
I feel like the question itself may not have been flawed. I would be interested to see you run this test again without the picture of the Super Nintendo displayed. My instincts tell me that this is a flaw in the test, and not the question itself. The question is clear, but the image throws people for a loop - plus the fact that people are expecting the Verify app to be used for memory tests.
This is a great example of how a form or process can confuse users if things aren't clear or they deviate from their expectations. Designers have to be aware of this and create products that not only present a message that is clear but also fit within norms and expectations.
Wouter Bron (ZURB) says
I agree with the point above. The question is fine, and it would work without the image. But in the current flow, you ask the question (List your five favorite Super Nintendo games), you show the picture, and then you ask another question: What can you remember?
Change that last question to: Now list your five favorite Super Nintendo games.
And your test perfectly makes sense.
Andreas Nurbo (ZURB) says
One problem is that the Super Nintendo console changed appearances. My first reaction was. Is that an Super Nintendo? So using image was probably a bad idea.
http://www.emuasylum.com/images/systems/Super%20Nintendo.jpg