Formative and Summative Usability Testing

When is the best time to do usability testing when developing a new product or service?

When you consider the place of usability testing in the overall product development process, then there are two main types of usability test: formative testing and summative testing (sometimes usability testing is referred to as user experience testing).

Formative Usability Testing

Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process and – if applied early in the design process – provides valuable insights of where users have difficulty reaching their user goals with the product (website, mobile app, desktop GUI design, hardware) or service.

In a typical project, we conduct two formative usability tests: one in the early concept stage where the UX design may be captured as a paper prototype with no working functionality. The paper prototype would contain wireframes of the UX design concept with several key usage scenarios supported in this version of the concept. The approach allows for a validation of the workflow and validation of initial UX design decisions around navigation, terminology and rough layout.

A second formative usability test happens when early versions of the software and hardware are available to catch issues right at the beginning of the implementation phase.

Summative Usability Testing

Summative usability testing is a Quality Assurance (QA) type of test usually performed later in the product development process. A similar usability test protocol is used as in the formative usability testing but now this setup is used to do formal user acceptance testing before the product is released to the target audience. Things, like think aloud protocol and hints for the user when they get stuck, are left out in this kind of testing. The pass/fail metric is the most important measurement in this test.

A summative test uses additional metrics of users’ success to assess whether the product meets those success metrics and can be released from a user experience perspective into the market.

Recommendation

I recommend to start doing formative usability tests as early as possible in the design process, right when the first wireframes are created and enough user experience design is done to support 5-8 workflows/tasks. You’ll be surprised how much can be learned at that point already!