The Magic Number: A Usability Testing Question

Running a usability study does not need to be an expensive endeavour since you are most likely looking to gain insights into a design rather than run a double blind study for FDA approval.  While the exact number may vary depending on the exact type of study, for usability studies the rule of thumb is that you can get close to the maximum number of meaningful insights at minimal cost with at least 4-6 participants per user persona, if the behaviours and tasks are unique for each persona or user segment. This relationship is illustrated in Figure 1.

What….only 4-6 participants…..how will you be able to prove that the problems encountered by your participants represent problems for a larger population? Unlike quantitative testing where the objective is to collect performance metrics, like time on task, which require a minimum of 20 participants, but potentially 50, usability testing is primarily qualitative. Participants in a usability study are asked to perform exploratory tasks which can expose a variety of design issues, but the lack of constraints on user freedom make performance calculations difficult, if not impossible.

While you may not be able to predict performance for a wider population at the end of a qualitative study, you will find trends that can apply to a larger population. If you recruit correctly, the participants that complete the study are representative of the larger user population, and if you find the 3 more participants struggle with the test, then target users will likely encounter similar difficulties.

Early research by Nielsen and Landauer (2000) found that with 5 participants you can find 85% of the usability issues. Running more participants will find uncover additional issues but may not be worth the added recruitment and incentive costs. Any critical insights should be readily evident in the initial 5 participants.

Another consideration is that the goal for a usability test is not to find all the concerns but to identify the critical or most common issues. You could test one group of 6 and find 75 issues, and then run another group of 6 and get 85 issues with only some issues overlapping between groups. You may not have the time and resources to try to tackle all the issues, and fixing some issues may introduce something new that will need to be tested and validated.

You should identify the top 5 -“ 10 concerns and once they are fixed, you should consider running another test to confirm that the previous concerns are no longer exist, and identify the next set of issues to tackle.  So, if you have a limited budget to recruit 15 participants, consider running 3 rounds of 5 participants and apply any insights gained between testing rounds.