Hitchhiker’s Guide to Notable

Navigate the Guide

Tests

Tests lets you quickly create design tests to collect feedback from a wide audience or your team.


Choosing a Test Type

Notable Tests give you 10 different types of tests to choose from. Each test type will give you different results and can be used in different situations.You can start a new Notable test on your dashboard, within a project, or from the top nav.

Start a Verify test

Preference Test

Use a Preference Test when you’re trying to make a decision about two versions of the same design. Avoid testing options that are too dissimilar — test takers should be able to understand that they are looking at two ways of representing the same functionality or content. Preference tests require at least two images — a control, and a variation, so get uploading!

Verify Notaball!
Notable tip!

Upload additional variations for more comprehensive results. Learn more about Multivariate Testing >>

Click Test

Create a Click Test to learn where your customers will click on a page or screen. This is a great test to create when you’re trying to figure out where your users will click to accomplish a specific task, so for best results make sure to include a brief introduction. Upload an image and add directive text or a question to give your testers some information or to help them understand the context of the test.

Multi Click Test

What’s better than a single click test? A Multi Click Test! Create a Multi Click Test when you’d like to understand where your users will click on pages in a flow.

You’ll need at least two screens to create a flow. To create the flow:

  1. Upload your first image
  2. Click Add Screen
  3. Upload the second image
  4. Repeat!
Create a multi-click test

Add directive text or a question to give your testers some information or to help them understand the context of the test. When you’re done adding screens, select Save and Select test Additions.

Yes / No Test

Create a Yes / No Test when you’d like to test whether users will click where you want them to. Make sure to include clear introductions to help set the context and objective for your tester!

Upload your image then choose up to three hotspots that represent a successful click. For example, if you want your users to click a big Buy Now button, create a hotspot around that button. And if you’re not quite happy with the hotspots, you can always reset and start again.

Create a yes/no test

Memory Test

The Memory Test is a great way to see what your users remember about your page. The image will be displayed to each tester for 5 seconds, so make those 5 seconds count! Make sure your introductory test sets up the right expectations before they see the image.

After you’ve uploaded your image, make sure your users know what they’ll be expected to do by giving them some instructions. This will be displayed to them for 5 seconds before the test begins.

Label Test

Having a debate with a colleague about whether to label your button “Send” or “Submit”? Label Test to the rescue! Create a Label Test and let your users tell you what they think UI elements should be labeled.

Make sure the UI component you’re asking your testers to label isn’t getting crowded by a lot of other elements. Make sure there’s enough white space around the component so that there will be no confusion as to what the tester is supposed to label.

When your image is uploaded, you’ll be able to add labels. These are the blank labels that your testers will enter text into when they take the test. If you made a mistake placing labels, don’t worry — there’s a handy Reset button that will remove all labels on the image and allow you to start over.

Add up to three labels to your test

Annotate Test

If you’re looking for freeform feedback from your testers, the Annotate Test is the way to go. This will allow your testers to create freeform annotations on the page and add their feedback.

Annotate tests can be a little more involved than other test types, so we will always include a brief introduction to how the tester is expected to give feedback. Make sure your testers know what kind of annotations they should be adding, by providing clear instructions that will appear before the test begins.

Mood Test

Create a Mood Test to find out how your users feel about pages and interactions. Request gut reactions here. This is meant to be as simple a test as possible, and a great way to test the basic emotional effect of your design work.

These are the options your testers will have to select from:

Choose three moods from eight options

Memory Question Test

Create a Memory Question Test if you want to ask your users specific questions about what they remember from a screenshot, instead of letting users give freeform feedback about what they remember.

This test allows you to customize the memory test a little bit more by adding your own question. Keep this question simple — your testers will try to remember a lot of information in a short amount of time, so keep the question short and sweet.

Add the question that you want to ask your users after they see your screenshot. You can get super specific and add multiple questions by clicking the Add Question button.

Create a memory question test

Question Test

Create a Question Test when you need to ask your testers a specific question. This test allows you to create a simple single-question survey. Hint: You can create a multi-question test by setting up a Linked Test with several question tests ;)

You can choose between 4 different types of responses including:

  • Free Response: a free form response from the tester.
  • Likert or Net Promoter Scale: to learn more about these tools, what they are, and how to use them, you can take a look at this example.
  • Multiple Choice: giving several choices of responses for the tester to choose from.

You can also attach an image to your question test, if that would help setup your question.

Add an image to your question test

Linked Test

Linked Test option lets you string a series of tests together, so you could distribute them as a batch. This is a great option when you're trying to gain insight into a larger design initiative and want to see the results in one report.

To create a Linked Test, choose Create Linked Test as your test type and setup the common settings from your tests. Then, you’ll be able to design any number of tests by adding them to your linked test.

Add test to Linked Test.

Design tests within a Linked test the same way as described above. Then choose to either Save and Add Another test or Save, when you’re finished.

Verify Notaball!
Notable tip!

Although there's no limit on how many tests you can include in a Linked Test, longer tests can be overwhelming and users may dropoff. If your study requires many tests, we recommend setting up several Linked Tests with 5-7 tests each.


Multivariate Testing

In multivariate tests, when someone takes your test, one of the image variations will be loaded at random. Our algorithm selects a variation at random while still ensuring an even distribution of testers to each of the variations.

Almost every test in Notable can be run as a multivariate test:

  • Click: up to 3 variations
  • Multiclick: up to 2 variations per screen
  • Yes/No: up to 2 variations
  • Memory: up to 2 variations
  • Label: up to 3 variations
  • Annotate: 1up to 2 variations
  • Mood: up to 3 variations
  • Memory Question: up to 2 variations

Creating a multivariate test is easy. When configuring a new test simply click on the Add a Variation button. This will allow you to add additional images to test against your control and enable the multivariate test report.

Add a variation for a multivariate test

Considerations for getting results

Keep in mind that if you have a control and a variation, you’ll need twice as many testers to get the same results as if you just got results of a control. For 2 or 3 variations you need 3x or 4x respectively.

Multivariate Preference Test

The preference test is the only test that requires at least one variation. Since the test itself is comparing two images, it is by definition multivariate. Since you are comparing a control to variations, the control image is always shown. For example, if you have one control and two variations, there are exactly two possible match-ups that people might see. Control and Variation 1, Control and Variation 2. Testers never NOT see Variation 1 and Variation 2.

Position bias with the Preference Test

There is a bias introduced when showing an image on the left side or the right side of the screen. To reduce this bias, the position of the control image is randomly selected during the test. For example, if you have a control and two variations there are exactly 4 configurations that will be shown: Control and Variation 1, Variation 1 and Control, Control and Variation 2, Variation 2 and Control. These different positions are not reflected in the results since they are only done to reduce bias.

Multivariate test summary results

On the summary page you will see the results of your control and be able to compare it to the results of your variations. Demographic filtering works for multivariate test and effects the results of all the variations. You will also be able to see the number of results that you received for each of the variations.

On the Raw Data tab, you will be able to see every results you received for the test. Each result will be marked with variations each user saw during the test.


Test Setup & Options

Importing Screenshots

All test types, except for Question Tests, require an image to be uploaded. Screenshots of your prototypes or site can be uploaded by choosing the Upload an Image tab in the Select a Screenshot section of test configuration.

If you want to capture a live site, choose the Capture by URL option to capture a screenshot of a public website. Optionally, a service like Ngrok can be used to setup a secure tunnel to expose a local server so that it can be captured within Notable.

Languages

Notable tests can be created and delivered in 12 languages! All language options were developed by customer request and have been tested by native speakers. You can choose to run the test in the following languages:

  • Arabic
  • Chinese
  • Dutch
  • English (default)
  • French
  • German
  • Hungarian
  • Italian
  • Japanese
  • Polish
  • Portuguese
  • Russian
  • Spanish

For tests in another language, all default and standard messaging is already translated and will be presented to your testers in the language you select. However, if you turn on Custom Message option for Intro, Thanks, or Demographics sections, make sure to add your message, otherwise we’ll display the default message shown in the textarea.

Write your message in your language of choice

Your testers will see the test in your selected language. The results will show whatever feedback message the tester leaves on the test. We don’t translate or alter the tester response in any way.

Introduction Page

The introduction and thanks pages are optional additions to your test that allow you to brand and provide additional instructions to your users before and after they take the test. To enable the Introduction Page, toggle on the checkbox next to Introduction Page. This will allow you to then upload a custom logo and provide introductory text to guide your users or welcome them.

Add an intro page to your test

Notable customers have used the introduction section of the test to host an assortment of custom content from NDAs, to an explanation of what the users feedback will be used for.

Thanks Page

Just like with the custom introduction pages, custom thanks pages allow you to further brand your test with your company’s logo and custom copy. To enable the Thanks Page, toggle on the checkbox next to Thanks Page during test configuration. This enables the Custom Message and Returning URL options.

Add a thanks page to your test

Toggle the Custom Message option and enter custom text for your testers to see once they have completed the test. If demographic information is enabled, this text will appear on the demographic page. If a custom message isn’t provided, the default message will be used; “Thanks for taking our Notable test! We’ll use the results to improve the design of the site.”

Enabling the Returning URL option allows you to specify a custom redirect at the end of the test. If the tester originally came from your site with the Notable Tests Bar for instance, redirecting back to your site provides for a smoother transition out of testing.

Please provide a Fully Qualified Domain Name (FQDN) and path to the page you would like to redirect to in the input. Relative paths will not work. The default return URL is zurb.com/notable.

Collecting Demographics

Collecting demographic data will help you gauge how certain types of users respond differently to your design. You can also use this information to filter out those you believe do not fit into your target market after your test has been filled.

You can collect demographics from your testers

There are eight types of information that can optionally be collected during testing:

  • Name
  • Twitter Handle
  • Email Address
  • Phone Number
  • Gender
  • Location *
  • Age
  • Company

*Location information is collected based on IP Address unless the result is a paid result. Paid result locations are specified by the user and are more reliable. Location information is always collected.

NOTE: Purchased test results include location, age and gender automatically, even if demographics is disabled.

Enabling the Custom Demographic Message checkbox allows you to display custom text on the Demographics Page of your test. Use this section to explain what will be happening with the user’s information. Users will be much more receptive to filling out information if they know what’s happening with their information.

If you don’t provide a custom demographic message, a default message is displayed: “We’d appreciate if you could take a minute and fill in these basic demographic questions - don’t worry, nothing nefarious.”


Defining the Audience

This step of the test setup determines who gets shown your test. Here you can choose whether your test is opened or closed and what devices testers should use when they take your test. You can also choose to add your test to the Notable Bar and embed it on your website.

Open Tests

Choosing the “Open Test” option makes your test available to the Notable testing community and provides you a free way to get additional responses.There is no way to determine who will take your test when choosing this option. This does not preclude you from sending your test out to specific test takers as well. However, all data is aggregated in one large pool, so keep this in mind when using open tests with your own user testing pool.

Closed Tests

Choosing the “Closed Test” option keeps your test private and generates a tokenized URL that makes it more secure. Closed tests are a great choice for testing content that you are not willing to disclose publicly. Bear in mind that with closed tests, it is up to you to share your test URL with your users. Some popular choices among Notable test creators are sharing on Twitter, sharing directly with their user testing pool, purchasing test results directly in Notable (fulfilled through Enroll), or embedding in the Notable Tests Bar.

Notable Tests Bar

Choosing the Add to Notable Tests Bar configuration will allow your test to be displayed on your homepage as a link in a bar at the top of your website. Customize and add the Notable Tests Bar to your site by going into Account Settings in the main navigation.

You can also customize the message and appearance of your Notable Tests Bar.

Choose your devices

You can further specify the types of users that can take your test by device. Choose whether you only want mobile users, tablet users, or desktop users to see your test. This is great if you are testing different layouts for different devices.

If you leave devices unspecified, users will be able to take your test on any device.

Choose the devices your test can be taken on

Finalize Your Test

The final step in test configuration is choosing how long your test should run, adding a title and description, and whether or not your test should go live immediately.

Test Duration

You can choose to have your test live anywhere from 10 minutes to 1 month. Shorter time frames are great choices if you are trying to limit the sample size and your potential audience is very large. Longer time periods are great for large samples and help average out your results.

If you have purchased results, we recommend setting a longer time frame – 1 day or 1 week are good options.

Test State

If you are done configuring your test and are ready to collect responses, you can choose the Live option. Your test will immediately be available for testers to take. If you choose the Paused option your test will not be accessible to testers.

Make sure that your test is live to receive paid test results. If your test is paused and you have purchased test results, your test will no not be accessible to your paid testers.

You’ll see the indication of your test state on the report page.

Make sure your test live

Test Reports

All Notable tests have comprehensive reports that give you both a summary view of your results, as well as the details of each and every response.

Report Summary

This result summary let’s you look at all your results in aggregate. The report is slightly different for each test but have many common components.

Most of the tests show a black and white version of the image to make it easier to see the results that are overlaid on the screen. You can toggle the full color version by clicking on the color wheel icon

See the screenshot in color or monochrome

For clicks tests, if you want to hide the hotspots and get a closer look at the image behind them then you can uncheck the Show Clicks checkbox.

See the clicks

Results counts

A standard (non-multi-variate) test will have three numbers detailing how many people have taken the test. Views, responses, and tester data.

  • Views count the number of people that have clicked on your test. We use cookies to make sure that if one person views the test multiple times it’s only counted as a single view.
  • Responses count the number of people that have completed your test. Once someone has completed a test they cannot take it a second time on that same browser.
  • Tester data counts how many people provided their tester data after completing the test. If a person completes the test but closes their window before entering their tester data, the responses is still counted as valid.

These numbers are shown here to let you quickly see how effectively you’ve engaged your audience. A high view to response ratio might indicate that you’re not providing a compelling reason for people to take your test, creating a intro page might help with this. You can also add a custom note on your demographics page if you have a low tester data to response ratio.

Filtering results

If you choose to collect demographics at the end of your test then you can filter your summary data by those demographics.

Clicking on the demographics listed below the summary will update the results to reflect that filter. You can apply multiple filters at a time to zero in on a specific segment.

Filter your results

Demographics

If you choose to collect demographics on your test then the demographics tab will give you an overview of the demographics of your respondents.

You can download a comma separated values (.csv) file containing these results if you want to import them into a spreadsheet or other system for tracking and comparing to different tests, or creating a custom visualization for a presentation.

One thing to note is that while most demographics need to be entered by the test taker, location is automatically collected. Because of this you might see more demographic data on location than other demographics. The location is determined by looking at the geographic location of the testers IP address. This provides an approximation of their location, not an exact address.

Raw Data

Every single result in a Notable test is individually tracked. By clicking on the Raw Data tab you can walk through these results.

Each test has a slightly different representation in the raw tab to give you a sense of what exactly the user submitted. You can also see the demographics entered for each result.

If you purchased test results you can review them one by one in the raw tab. If you’re not satisfied with a result click on the flag link to mark the result as invalid. The result will be deleted and a new tester will be automatically queued up to replace it at no additional cost to you.

You can download a comma separated values (.csv) representation of your test results from the Raw Data tab. This lets you manipulate this data in another system like a spreadsheet or database to gain further insights.


Notable Tests Bar

The Notable Tests Bar allows you to add a link to a group of tests on any website using a simple Javascript snippet. Use this when you want to get quick feedback from your existing customers.

The Notable Tests Bar will appear at the top of the viewport on a live site. You can customize the text that will appear to customers, and they can also easily dismiss the message if they don’t want to take the test. You can choose to add any test to your Notable Tests Bar during the test setup process.

Add a Notable Tests Bar to your website

Add the Notable Tests Bar to Your Site

During the test setup, choose Add to Notable Tests Bar option in Define Audience step. Once your test is added to the bar, go to Edit Account page in Notable to grab the code snippet and paste it into your site HTML.

Once you’ve copied the code snippet from the account page, paste it into the <head> element in your HTML.