Posted on: January 9, 2014
Ph.D - Principal Experience Architect
The Magic Number: A Usability Testing Question
Running a usability study does not need to be an expensive endeavour since you are most likely looking to gain insights into a design rather than run a double blind study for FDA approval. While the exact number may vary depending on the exact type of study, for usability studies the rule of thumb is that you can get close to the maximum number of meaningful insights at minimal cost with at least 4-6 participants per user persona, if the behaviours and tasks are unique for each persona or user segment. This relationship is illustrated in Figure 1.
What….only 4-6 participants…..how will you be able to prove that the problems encountered by your participants represent problems for a larger population? Unlike quantitative testing where the objective is to collect performance metrics, like time on task, which require a minimum of 20 participants, but potentially 50, usability testing is primarily qualitative. Participants in a usability study are asked to perform exploratory tasks which can expose a variety of design issues, but the lack of constraints on user freedom make performance calculations difficult, if not impossible.
While you may not be able to predict performance for a wider population at the end of a qualitative study, you will find trends that can apply to a larger population. If you recruit correctly, the participants that complete the study are representative of the larger user population, and if you find the 3 more participants struggle with the test, then target users will likely encounter similar difficulties.
Early research by Nielsen and Landauer (2000) found that with 5 participants you can find 85% of the usability issues. Running more participants will find uncover additional issues but may not be worth the added recruitment and incentive costs. Any critical insights should be readily evident in the initial 5 participants.
Another consideration is that the goal for a usability test is not to find all the concerns but to identify the critical or most common issues. You could test one group of 6 and find 75 issues, and then run another group of 6 and get 85 issues with only some issues overlapping between groups. You may not have the time and resources to try to tackle all the issues, and fixing some issues may introduce something new that will need to be tested and validated.
You should identify the top 5 – 10 concerns and once they are fixed, you should consider running another test to confirm that the previous concerns are no longer exist, and identify the next set of issues to tackle. So, if you have a limited budget to recruit 15 participants, consider running 3 rounds of 5 participants and apply any insights gained between testing rounds.
Ph.D - Principal Experience Architect
Daniel is dedicated to creating and improving software, website and interface designs to provide an exceptional interaction among users, their environment and their tools. To achieve this goal, Daniel applies knowledge and techniques from engineering, computer science, psychology and anthropology learned while completing a Doctorate in Systems Design Engineering from the University of Waterloo with a focus on human factors, cognitive ergonomics and user-centered design. Prior to joining Akendi, Daniel was the user experience designer for Brock Solutions, and was instrumental in the design of software used by airports and airlines around the world. In addition, Daniel has provided independent human factors consulting services for several large and medium enterprises.
Sign up for our UX Blog
Don't miss an article! We'll notify you of each new post.