Blog Image
Dominira Saul

Dominira Saul

Akendi Alumnus

Usability Testing is NOT Public Opinion Research

This week it was brought to my attention that the Canadian Government was reviewing the standing offers that pertained to Public Opinion Research. The big news was that under Public Opinion Research there was now a new category for Usability Testing. When I heard that – my eye started to twitch.

You might think that having a vehicle such as a standing offer, whereby the government recognizes usability testing as a service that is worth paying real money for would be a good thing. It is. The problem is that it’s categorized under public opinion research.

Before I launch into my tirade, let me say that I have the greatest respect for POR… some of my best friends work in POR (no, not really).   Honestly, I see the value of focus groups and surveys and the many tools that POR uses. Many of those methodologies can be used for user research in UX, but listen carefully… Usability Testing is Not Public Opinion Research.

The fact that usability testing is now under the purview of Public Opinion Research is worrying for me as a supplier of UX services.

While it is a good thing that Usability Testing is on the government radar, it is lumped in with polling, marketing and POR in terms of procurement and that is fundamentally wrong.

First of all, usability testing is a specific UCD methodology.  Those who aren’t aware of user centred design or the methodologies for user research, as opposed to market or customer research, tend to group a wide range of research and testing methodologies under the umbrella of Usability Testing. I imagine it’s because that is the only terminology with which they are familiar.

Often methods such as Usability Walkthroughs, Contextual Inquiry, Ethnography, Heuristic Evaluations, Cognitive Walkthrough, Card Sorting are mistakenly categorized as Usability Testing. This is detrimental to those who conduct such research as well as to those who want to gain access to experts in those particular methods.

True usability testing has nothing to do with opinion.  The entire purpose of usability testing is to remove opinion and conjecture from the design process.  Usability testing is designed to observe and measure user performance and behaviour… not opinion.

Common metrics in a usability test are task completion time, optimal navigational path, error rate, rate of error recovery etc. These are all quantitative, empirically measurable metrics; again… there is no opinion involved.

Gathering Qualitative Data

There are two instances when qualitative data is typically gathered.  Firstly, when a participant is asked to explain what their thought process was in relation to certain actions.  This can be done during a usability test session (known as a think aloud protocol) or after the action is complete.  This is not opinion.  It is a participant explaining the thought process associated with the observed action.

The second instance where qualitative data would be gathered would be post task, or post test questions when participants are usually asked to rate ease of use, or confidence in completing the task.

These questions are important because:

  • In a discretionary use scenario, user satisfaction is most probably the largest single key factor, which will influence the users’ decision whether or not to continue with the software.
  • In a mandatory use scenario, poor satisfaction leads to absenteeism, fast staff turnover, and unrelated complaints from the workforce.
  • The subjective measurements complement data from efficiency and effectiveness (empirical) measures.
  • Subjective measurement usually produces a list of satisfying and unsatisfying features, which is especially useful if testing is taking place during development.

Many government agencies seek to comply with the Treasury Board of Canada Secretariat’s Standard on Web Usability, the goal of which is to ensure that users can effectively and efficiently find, understand and use the information and services provided through websites and Web applications.  Categorizing Usability Testing under Public Opinion Research will encourage those who conduct public opinion research to use methods that are more closely aligned with the kind of research they use in their own field, leading to less rigorous testing and poor results.

According to the ISO definition of Usability, Effectiveness and Efficiency are measured empirically.  There is no stipulation in the TBS requirement for Satisfaction.  Satisfaction is the part of the ISO definition of Usability that corresponds to the collection of subjective qualitative data.  In short, there is no requirement under the TBS rules to collect anything other than Empirical data, which to me means that usability testing of government websites has nothing to do with public opinion.

Support legitimate Usability Testing

In conclusion, it is clear that the Treasury Board Secretariat believes that usability of online government services is important.  It is my sincere hope that the current crop of POR standing offers will be used to bolster and support efforts for legitimate usability testing instead of becoming an obstacle by adding layers of complexity and red tape to the process of procuring usability testing services or worse yet become a vehicle whereby those who are unqualified, do not practice or understand the methodology, peddle some watered down and bastardized version of what they call “usability testing” to an unsuspecting, but potentially lucrative customer base.

Tirade over.

Dominira Saul

Dominira Saul

Akendi Alumnus


My eyes are twitching too!
“Usability” seems to be the buzzword of the day, with alot of folks getting in on the act who do not understand its meaning, or the rigours of the practice of user-centred design. I’ve seen statements of work that specify focus groups as the way to validate information architecture and wireframes.
More folks in govenment need to take your certification courses – in particular those involved in POR or in application development.
best regards,

Thanks for the comment Mary. I’ve seen those RFPs too. Right now there is a Request for Industry Comments (RFIC) going on for the new POR standing offer categorizations. Maybe if we all lend our voices we’ll see some change.

Time limit is exhausted. Please reload CAPTCHA.

Learn how your comment data is used by viewing Akendi's Blog Privacy Policy

Related Articles

About Akendi

Akendi is a human experience design firm, leveraging equal parts experience research and creative design excellence. We provide strategic insights and analysis about customer and user behaviour and combine this knowledge with inspired design. The results enable organizations to improve effectiveness, engage users and provide remarkable customer experiences to their audiences.