Making the natural ‘natural’?

Should you test in-person or remotely? What to choose when there are so many options? If the question was simply about the quality of the test session, I would say in-person, hands down.

Just think about the difference between a Skype call with friends or family overseas versus sitting down in the living room with them. It isn’t just about the questions we ask but how the environment influences our behaviour.

The more natural the environment is to the behaviour at hand, the more likely we will behave the way we would in a non-test environment.

Many companies are understanding the importance of this. One of our Telco clients has built a fully furnished apartment where users can test various products, as if they were in their own home -� or perhaps their ‘ideal’ home as the size of the televisions (yes plural) are a TV lover’s dream. A consumer electronics client of ours has designed an entire home lab that is used to test a variety of products -� the rooms are all fully mic’d and recorded.

Participants can act ‘naturally’ as they would normally with your products, almost forgetting after a while that they are being recorded -� not too different from our favourite reality shows.

The downside of moderated lab testing is that you are limited to those people that can make the journey.  In many cases, this is not an issue as you are testing user personas whose motivations, skills, needs, etc. may have no bearing on location.  However, sometimes this is not the case.

Healthcare, in particular, is one of the more challenging things to test, not only because it is difficult to recruit surgeons, for example, but it might be extremely difficult to do this in the context in which they work (if they are not coming to your own lab).  For example, you might disturb the workflow, there might be issues with confidentiality, access to Doctors, interference with equipment, and the list goes on.

Retrospective testing can be used as a workaround for the above scenario.  This involves having the participant record their experience in their context and then at a later time reviewing the recording together with the researcher. Though less ideal than in person, it does allow the researcher to probe more into the things that the participant might not have elaborated on, or in fact noticed.

One of the courses I teach during Akendi UX Training is Usability Testing.  I often have students asking me whether there are benefits to remote un-moderated usability testing. Certainly the ability to reach a large number of people with no limitations to scheduling and distance, and more context specific (no need to create a simulated lab) is a great bonus.  However, it often lacks the depth that a moderated test allows. People will answer your questions, but if they misinterpret it, there is no one there to help them get on track.  If they say “huh?” but do not elaborate ‘why?’ there is no way of knowing.  The list goes on; and this does not even speak to the potential issues with the type of access to technology that is required from each user.

That said, there might be some validity in pairing an un-moderated usability test with a retrospective test. Similar to asking people to fill in an online survey and then asking for a follow up call, web-ex can be used to review an un-moderated usability test to get at the ‘why’ whilst maintaining the user’s context of use.

I still will vouch strongly for an in-person moderated usability test, as the value of information is incomparable. But when there are limitations with access and context, and technology that is continually advancing.

It is important as a researcher to continually review and challenge your research methods to ensure that you receive the high quality results that will help to enhance the overall user experience of your product or service.