My introduction to the world of human factors and usability testing was through a book entitled, “Set Phasers on Stun: and other true stories of design, technology, and human error” by Steven M. Casey. Set Phasers on Stun is a collection of short stories where the misalignment between the design of technology and user behaviour led to disaster. The title story was an incident involving a system that focused radiation to treat cancer. The device operator, an experienced nurse, accidentally triggered an operational mode where the lead safety shield between the patient and the radiation source was removed. The patient became aware something was wrong when the treatment resulted in pain, but was unable to alert the nurse because the operators’ room was down the hall from the treatment room and there was not system to communicate between the two. Meanwhile the nurse was being presented with an unfamiliar error message. Since the error message, “Error 459: Treatment Failure” did not clearly describe the source of the problem, the nurse interpreted the error as an indication that the treatment did not start, and repeated the treatment process, resulting in several deadly dosages of radiation.
To avoid outcomes, similar to Set Phasers on Stun, the US Food and Drug Administration released a draft document on applying human factors and usability engineering to optimize the design of medical systems. While Health Canada has not yet issued any guidelines around usability testing, it is safe to assume that they are on the way.
Typically, medical standards are designed to reduce the risk of electrical, mechanical, chemical, biological, thermal and radiation hazards. By supplying guidelines encouraging a user centered design process and usability testing, the FDA is recognizing the need to reduce the risk from hazards due to usage. Device manufacturers are expected to conduct a comprehensive risk assessment to determine if incorrect or unexpected use could cause death or serious injury to the device user or the patient and demonstrate that the intended users would be able to use the device effectively and safely under realistic conditions. This is achieved though rigorous usability testing. If the idea of rigorous testing raises concerns, something to keep in mind is that usability tests are not clinical trials. Usability testing requires fewer participants (sometimes only 4-5 are enough, for summative tests 15 participants and more), take less time (can be completed in a couple weeks), and can be performed early in the design process to identify user-related problems that can be addressed easily and at lower costs.
For effective and valid usability testing, it is important to identify and understand the intended users, the tasks they must perform with the system, and the environment in which they perform the tasks. The participants should be representative of the intended users. Job title and role, knowledge, experience, age, functional capabilities (physical and cognitive) are just a small sample of user characteristics that you may need to take into consideration. If your system is going to be used by oncologists, you should be testing with oncologists. Next, study participants should be allowed to interact with the device under realistic conditions and in a natural manner, without interference or influence by the study administrators.
However, usability testing of medical system cannot be treated in as light handed manner as maybe found in usability testing of consumer systems. Protocols must be more clearly defined and adhered to due to challenges with participants, and the testing environment. Testing of medical devices may require the recruitment of participants that are physically or cognitively vulnerable. The design of the usability testing must ensure the privacy of the participant is protected. The testing environment must be representative of the typical context of use. For medical systems these environments may need to be; sterile, stressful, noisy, complex, uncomfortable (like the back of an ambulance), and other adjectives that may not be replicable within a lab. Finally, reporting the results of a usability study will be more complex and should follow the common industry format (CIF) standards like NISTIR 7742.
With end-users (patients, doctors, nurses, hospital staff) performing realistic tasks, in the right context, usability tests of medical systems are a quick and effective approach to reduce usage risk, and will result in cost savings by reducing development time and liability. But the approach to usability testing cannot be haphazard, and careful planning is required.
Daniel Iaboni is a Senior Experience Architect at Akendi, a firm dedicated to creating intentional experiences through end-to-end experience design. To learn more about Akendi or user experience design, visit www.akendi.com.
Akendi is a product strategy, user experience design and usability research firm. We are passionate about the creation of intentional experiences – whether those involve digital products, physical products, mobile, service or bricks-and-mortar interactions. We work shoulder-to-shoulder to optimize the experiences you deliver. Akendi Corporate Overview (PDF).
Experience Thinking innovation firm in Product UX Strategy, User Experience Design & Usability Testing for Companies: Toronto, Ottawa, Montreal, Vancouver, Canada.
T: +44 (0)20 35982601
22 Highbury Grove
London, N5 2EF