The Sweet Spot – Participant Numbers in Qualitative UX Research

Being a UX designer I conduct a lot of qualitative research, from developing personas and journey maps to validating websites and products. There is a common theme that comes across, especially when validating design; some clients feel we did not test enough people. While we have recruited the number of participants based on budget, time and method, I still run into the remark “you only tested ‘x’ amount of people?”. “Should we not be testing more?”.

How do we combat this question on the spot? What if the client keeps pushing for more testing sessions on a limited budget? How does the number of personas play into the number of participants recruited for research and testing studies? Now that I have your attention, let’s jump in.

Personas and Journeys

How many interview participants do you need to validate personas and journeys? In my opinion, this depends on how complex the system is and how many different tasks and types of users are generated from the stakeholders in an initial workshop. For example, if the stakeholders identify 9 personas in their system, you will need more interview participants to validate these personas and their journeys in that system.

Before interviews even begin though, you may be able to consolidate personas before recruiting. Sometimes I find the clients think of personas as different job titles/ positions. While this may work in some instances, we know that personas are based on behaviour and tasks and not solely on demographics, or even necessarily job titles. So first, even before recruiting, I would suggest seeing if you can consolidate personas and notice if there is any overlap in the tasks of each declared persona.

Once the personas are consolidated to the smallest number without compromising on tasks and anticipated user behaviour, I would start the process of recruiting (or talking to a recruiter).

Factors that affect the number of participants

The number of participants recruited for a research study depends on the client budget, time constraints, number of assumed personas, and complexity of the system. If there is less time and/or money, the number of participants recruited will be smaller. Once budget and time are specified, this will help add clarity to the total amount of participants used for the study.

By catering to a client’s request for a large number of participant numbers, you potentially run the risk of saturation in data. The principle of saturation is that while data is still being gathered, few to no new insights are emerging in patterns. There is such a thing as over-recruiting. Qualitative research is based on patterns and not overall numbers, so there is no use in recruiting a large number of participants to validate a persona or journey.

The complexity of the system and variance between personas does have an effect on the number of participants being researched. Most academic research says between 20-50 participants is enough before reaching saturation within qualitative research, however other academic research says you can go as low as 10 participants to reach saturation depending on the system.

What is important to consider when recruiting is what qualitative research method are you using, how complex the system is, and how much variance there is between personas and the tasks they perform in their journey overviews.

In general, I recommend that recruiting 3 participants per persona is an ideal place to start when validating and growing a persona. This means that if you have 5 personas, 15 participants is a good initial number to bring to your clients for a discussion about research needs.

Numbers needed to validate a design

I often experience the client questioning the number of participants in a usability study (or other forms of validation studies) after the results have been given, especially if the results are not what was expected/ hoped for by the client. Once more the moment comes when you have to justify the number of participants recruited for a study.

Things like time and money similarly do affect the number of participants recruited in testing (just like in research) and this again is something you can speak to. But there always tends to be one particular person that says “we will pay you more — let’s test more people — this number of the participants you used does not reflect our user population”. At the end of the day, it may be tempting to take more money, but be aware that your work may become redundant.

Low numbers, high frequency

Nielsen and Landauer found in 1993 that by testing 5 participants, 85% of usability problems would be brought to light in testing. With this in mind, we can feel comfortable suggesting the low number of 5 participants for usability testing to clients when appropriate (when there are restraints on time and money).

However, some academics are vocalizing that 5 participants are not the minimum required in all circumstances, most saying 7-12 is closer to the ideal number. Just like in research, we need to be taking into consideration the complexity of the system and how many different variations of users there are (personas). From my experience in testing, I find saturation begins after 20 participants in most circumstances.

Conducting frequent qualitative testing to validate a product is the most beneficial way to keep the user experience at a high level, making sure all user needs are being met. If a company has a big budget, I would suggest spending it on frequent qualitative testing with a focus on different experiential topics, as opposed to getting more participants in a singular testing study. By conducting qualitative research intermittently, the nature of the insights is more current, detailed and topical, making a good return on investment for the client.

More is better?

UX designers and researchers alike need to fine-tune the communication to their clients around the idea that “more is better” when it comes to testing. We need to help break the old stigma that more participants in a study equal better results, as this is not always true. Let’s take the time to speak to our clients and reflect on what is really needed from testing and research to get value for the client, but also not waste time doing redundant work.