Tight budgets are nothing new in product development land. Going lean & mean to meet budget and time constraints happens all the time. The one thing that shouldn’t be avoided though is conducting a usability evaluation of your actual site or system before you go live. Here at Peak our preferred usability evaluation method is to conduct testing of a site or system with real representative users. As ‘expert’ user experience (UX) practitioners we are often asked to conduct an expert review of a site or system when budget or timing is tight. An expert review typically consists of a cognitive walkthrough and a heuristic evaluation. Expert reviews have their benefits but they also have their drawbacks. In this article we explore the pros and cons of expert reviews versus usability testing and draw on a recent project which had some unexpected outcomes which triggered this article.
Sometimes known a ‘discount usability method’ conducting an expert review involves using discount or desktop usability evaluation methods (UEMs) to evaluate a site or system using heuristic checklists, cognitive walkthroughs and/or guidelines.
We recently had a client approach us to assess a portion of a form based transactional system that had just been launched. They wanted to get a better understanding of any usability issues in their new online transaction. Our client only had a very small budget so our first reaction was to conduct an expert review of the system. We had difficulty justifying the expense and value of an expert review when on initial inspection there did not appear to be hardly any usability issues. The online transaction appeared to be nicely designed, flowed well and was mobile & tablet friendly.
To meet the client’s needs and budget, we proposed an alternative approach of ‘guerrilla’ usability testing. Because of its widespread usage of this type of online transactions by most Australian adults, we were able to conduct 10 X 15 minute test sessions with relative ease and source a wide enough range of participants from friends, family and our business partner with whom we share an office (no gorillas this time).
Expecting users to blitz through the online transaction, we were somewhat surprised to discover a number of usability issues due to the variety of participants each with their own unique situations, needs and mental models. A simple question asking: “How is the property occupied?” proved difficult for landlords to answer without being taken down the wrong product path.
Another question asking “Is there any business activity conducted from the home?” didn’t provide any way to distinguish between working from home and running a business entirely from a home. This was then further misconstrued through a follow up question “How many employees (other than household members) work at the insured address?” as the participant was still thinking about how his family based café business related to this question when in reality they only did minor book keeping duties from home.
These are just a couple of examples and in total we identified 20 issues which was quite unexpected.
Had we gone with an expert review the system we don’t believe we would have identified anywhere near the number of issues as we did when testing with real users each with their own unique circumstances and mental models.
Expert reviews do have their place and often as experienced UX consultants, when we are conducting testing of a site or system we identify a lot of the low hanging fruit and major usability issues before we even start testing. We often comment internally that we could easily have identified most of the issues without testing. The problem with this approach is that stakeholders and clients don’t necessarily listen to us when we told them and they would think that is just our opinion. When we conduct usability testing with real users, our clients are often happy that we finally provide them with the evidence for issues they have been telling the business about for years. It is very hard for business stakeholders to refute the evidence of real users or customers failing to complete tasks using your system, especially when they watch video clips of users suffering a poor user experience.
So in summary, expert reviews are quick and affordable as don’t need to recruit and pay users and can turnaround a desk top review quickly. However, they are best done by an experienced UX practitioner who has observed a lot of test sessions and understand interaction design principles.
However, the benefits of usability testing generally outweigh the benefits of expert reviews:
- less reliant on the evaluator’s knowledge (although experienced testers will pick up a lot more issues).
- are more accurate as they reflect real users attempting common tasks using your site.
- are less subjective and results are likely to be accepted by business stakeholders.
To learn more about our usability evaluation methods including expert reviews and usability testing why not attend one of our UX Design Courses.
Usability tip for an expert review:
In the absence of real users, try and determine actual user scenarios. Use these scenarios for the expert review rather than scenarios you’d do personally.
Usability tip for usability testing:
Record some sessions (if you’re not already) and review yourself! You’ll be surprised with how many little errors you make as a moderator, e.g. leading questions, body language hints, etc. These can be things to focus on improving in the next session.