A range of values that allow you to confidently estimate your untested population.
When giving an answer to a question, we are often asked: "On a scale of one to ten, how confident are you with this?" We then rummage through our mind to gather the data as needed to give an estimate on our level of confidence.
We can now give that question an actual numerical and mathematical value with the use of confidence intervals.
With confidence intervals, we can take tested data and apply a quick equation to find out an estimated range of probabilities with confidence.
The simplest explanation for confidence intervals is that they are the most likely range of an untested population average. These intervals are applied to your already collected actionable user test data and results to determine your sense of confidence to push forward with your products. If you make the most of your design decisions based purely on a hunch, you are missing out on the highly useful tool found in confidence intervals.
Calculating Confidence Intervals
If there is one thing that most people abhor, it is math. Don’t worry, we will try and keep it as simple as possible. We don’t want you to have an aneurysm.
Though the equation for calculating the intervals looks intimidating and daunting, when you break down all of the inputs to the equation it’s much easier to swallow. The only necessary inputs to figuring out your confidence intervals are: number of trials, number of successful rounds of trials and the confidence level.
Here is what your equation looks like to find your confidence intervals:
z= z-value that corresponds to confidence level (your confidence level + half the squared value)
padj = (n*p + z^2/2)/(n + z^2)
nadj = n + z^2
… and the last bit will find your confidence intervals:
padj ± z * sqrt(padj(1- padj)/nadj)
If that equation just floored you and you really worked up a sweat, don’t lose any sleep over it. If you would prefer to make it extremely easy for yourself, just use this handy-dandy confidence interval calculator.
With the equation to figuring out our intervals, you should know that there are three influencing factors that impact the breadth and look of our confidence intervals. They are the following:
We have already noted that this element is important in figuring out your confidence interval, but it can be adjusted to alter the reach of your intervals. Most confidence intervals are calculated at the 95% confident level mark. However, it can be calculated at any percentage you’d like. The confidence level determines how much of our estimate on the unknown or untested population will be wrong/right. An example may prove to be useful in gaining a better grasp of how these levels affect your intervals.
If we look at a comparison of two different confidence levels, using our lengthy algorithm, we can see how the confidence level will adjust the look of our confidence intervals. Let’s start by using an example of five test participants, and four successfully complete a task. Then by using 95% confidence level, we see that we have a confidence interval that stretches between a 36% and a 98% chance in the real world. However, if we use 99% confidence we will see that we have interval that spans between a 26% and 99% chance of seeing similar results.
Confidence levels may sound a bit confusing at first, but let the example sit with you for a bit and it should make more and more sense.
The higher the confidence level we have, the lower risk of our estimates being wrong. At the same time, if we increase our confidence level, we are increasing the span of our confidence intervals over our set of data. To make it even simpler, the higher a confidence level the more values will be added to our confidence intervals to avoid our estimates being wrong.
The variability of your test taking participants and their results factor into how your confidence intervals are spread. If you are testing a primarily tech-savvy group your confidence intervals may look high in your favor, whereas if you were to test a group that doesn’t use much tech or tech devices, the results and confidence intervals will look mighty glum.
Similar to variability in that the more subjects you have, the shape and look of your intervals will change. With sample size, though, the more subjects tested, the smaller the intervals become in contrast to smaller sample sizes intervals will be viewed much larger. Getting a more focused look into how your tested population influences your intervals may benefit your understanding.
Let’s go back to our lovely equation and look at how the sample size affects our confidence intervals.
To further express how differently the sample size alone affects the intervals, we will look at two different sample sizes that have the same result percentage. We will be looking at the ratios 4:5 and 80:100 and confidence level of 95%. We’ve already noted in this circumstance that our 4:5 ratio has a confidence interval between 36% and 98%. With our ratio of 80:100, we find our confidence interval drastically shifts to being between 68% and 88%. Though, with a larger sample size your confidence interval drops, you are narrowing your estimate to a more approximate location rather than spanning between the lowest possible percentage and highest possible percentage.
We have mentioned before that one of the best, cost-effective methods of testing is with five testers. If you’d like to put this theory to the test and want to get the most bang for your buck, you can use the same tools that we at ZURB use as well. Within Enroll there is a large pool of diverse, unbiased and willing participants for you use for your user testing needs that can be used in conjunction with two fabulous applications: Solidify and Verify — yes, we created these tools as well (forgive the shameless plug).
"Confidence Intervals, Give Me Sight Beyond Sight!"
The confidence intervals have an innate ability, much like the Sword of Omens, as they have the ability to help us see things which we normally could not see — an estimate of the whole population of end-users and how they would handle your design.
These intervals can help in shaping your proof points as well. Proof Points are described as being the specific pieces of evidence used to substantiate the unique claims found within key messages and positioning statements. If you are to say your product is "ahead of its kind in its market share" this proof point could be backed up with specific data and evidence to provide to your potential clients.
With these ideas and concepts tucked up your sleeve you’ve added to your arsenal a trick that will aid in producing the highest quality designs and iterations. Remember, you need to have the confidence in publishing your designs to be used by the end-users of the world. That’s where your knowledge of the confidence intervals helps immensely. The tools are all available to you — it’s just a matter of applying them to your work and reaping the benefits of your work.
More Web Goodness
A New Formula for Quantitative UX Decision Making
What Is a Confidence Interval and Why Would You Want One?
10 Things To Know About Confidence Intervals
Wikipedia: Confidence Interval
Video: Statistics is Easy - Confidence Intervals
How To Communicate Design Decisions To Clients?