Ask the wrong question and your results will be flawed. Whether you're asking an online survey question or conducting a face to face interview the question you ask can make or break your entire experiment or interview.
Recently, I had the idea to run a poll for the greatest Super Nintendo games of all time using a Verify Memory test. It seemed like a great fit, the actual test asked people for five games they remembered. The report grouped responses into the top five related phrases. I was surprised at the results.
More than half of the people that took the test did not follow the instructions.The plan was to simply instruct users to enter their five favorite Super Nintendo games into the form, rather than their memories of the screenshot.
Why is this? To put it generally, the test was poorly designed.
To be more specific:
- We asked a question about software (games), and then showed a picture of hardware (Super Nintendo)
- The survey did not follow the established pattern for a memory test, of collecting memories from an image
- The instructions were not attached to the form
- An image of a Super Nintendo was disimilar enough from the instructions to cause people to question what they were doing
- Images are more powerful then words, people followed the instructions implied by the image, rather than the explicit written instructions
Our takeaway from this fun little experiment is to think long and hard when phrasing a question. Whether it's Verify or a Q&A interview your question is what will make or break the results. When dealing with Verify tests make sure to:
- Have just one introduction/instructions screen
- Make your instructions a maximum few lines long
- Make the question/instructions as clear and concise as possible. You grandmother should understand them.
- Shoot the preview of your test to a few colleagues to see if they can understand and take your test without your help