A usability evaluation method in which one or more reviewers, preferably experts, compare a software, documentation, or hardware product to a list of design principles (commonly referred to as heuristics) and identify where the product does not follow those principles.
Heuristic evaluation falls within the category of usability engineering methods known as Discount Usability Engineering (Nielsen, 1989). The primary benefits of these methods are that they are less expensive than other types of usability engineering methods and they require fewer resources (Nielsen, 1989). The beneficiaries are the stakeholders responsible for producing the product – it costs less money to perform a heuristic evaluation than other forms of usability evaluation, and this will reduce the cost of the project. Of course, the users benefit from a more usable product.
The original justifying assumptions about the heuristic evaluation method presented by Nielsen (1989) and Nielsen and Molich (1990) were: the method is relatively inexpensive, evaluators do not have to be usability experts, there would be evaluations by “several” evaluators, and that when results are aggregated, the evaluators will find “most” of the issues identified by more expensive methods.
The literature presents mixed messages about the relative advantages and disadvantages of the heuristic review method. For example, two papers written in the early 1990s (Jeffries, Miller, Wharton, and Uyeda, 1991; Desurvire, Kondziela, and Atwood, 1992) compared the effectiveness of different usability evaluation methods, including heuristic reviews.
Jeffries et al found that heuristic reviews identified more usability issues than the other methods used in their study -- usability test, guidelines review, and cognitive walkthrough – when the results of all evaluators were aggregated (Jeffries et at, 1991). However, their definition of heuristic evaluation differed from method described by Nielsen and Molich. They used experienced usability professionals – “UI specialists study the interface in depth and look for properties they know, from experience, will lead to usability problems” (emphasis added) (Jeffries et al, 1991). Further, while heuristic evaluation identified more issues than usability testing, heuristic evaluation identified more minor issues and usability testing identified more major issues (Jeffries et al, 1991).
Desurvire et at (1992) showed that experts identified more usability issues than non experts, Desurvire et al’s finding supports the results reported by Nielsen (1992), that usability experts identify more issues than non experts, and double experts – usability experts who are also domain experts – find more issues than usability experts. Jeffries and Desurvire (1992) point out that to realize the full benefit of a heuristic review, the evaluators should all be experts, and the cost of multiple experts will increase the cost of the review.
Cockton and Woolrych (2002) reviewed “discount” usability methods from the perspective of a cost benefit analysis. They point out that actual user problems result from a complex interaction between the user and the system (Cockton & Woolrych, 2002). They believe that discount methods, including heuristic reviews, are too simple to accurately evaluate this interaction (Cockton & Woolrych, 2002). They concluded that these methods are so prone to error that the potential costs far outweigh the benefits (Cockton & Woolrych, 2002). Cockton and Woolrych (2002) recommend that these methods “should be cleared off the HCI store’s shelves.”
Molich and Dumas (2005) reviewed the results of Comparative Usability Evaluation 4
Jeffries and Desurvire (1992) point out that, if the results reported by Nielsen (1992) and Desurvire, Kondziela, and Atwood (1992) – that experts or double experts will find more issues than nonexperts – then heuristic evaluation becomes more of a Cadillac method than a discount method (Jeffries and Desurvire, 1992).
Cockton and Woolrych (2002) reviewed “discount” usability methods from the perspective of cost benefit analysis. They point out that actual user problems result from a complex interaction between the user and the system (Cockton and Woolrych, 2002). They believe that discount methods, including heuristic reviews, are too simple to accurately evaluate this interaction (Cockton and Woolrych, 2002). They concluded that these methods are so prone to error that the potential costs far outweigh the benefits (Cockton and Woolrych, 2002).
Heuristic evaluation can be used throughout the design life cycle – at any point where it is desirable to evaluate the usability of a product or product component. Of course, the closer the evaluation is to the end of the design lifecycle, the more it is like traditional quality assurance and further from usability evaluation. So, as a matter of practicality, if the method is going to have an impact on the design of the interface (i.e. the usability issues are to be resolved before release) the earlier in the lifecycle the review takes place the better. Specifically, heuristic reviews can be used as part of requirements gathering (to evaluate the usability of the current/early versions of the interface), competitive analysis (to evaluate your competitors to find their strengths and weaknesses) and prototyping (to evaluate versions of the interface as the design evolves).
Nielsen and Molich described heuristic evaluation as “an informal method of usability analysis where a number of evaluators are presented with an interface design and asked to comment on it” (Nielsen & Molich, 1990). In this paper, they presented nine usability heuristics:
This list, and later versions (for example, Nielsen, 1994; Nielsen, Bush, Dayton, Mond, Muller, & Root, 1992), are commonly used by many practitioners as the basic heuristics for product evaluation. However, there are other published lists of heuristics available, including Shneiderman’s eight golden rules of interface design (Shneiderman, 1998), Gerhardt-Powals research-based guidelines (Gerhardt-Powals, 1996) and Kamper’s lead, follow, and get out of the way principles and heuristics (Kamper, 2002).
Heuristic evaluation is not limited to one of the published lists of heuristics. The list of heuristics can be as long as the evaluators deem appropriate for the task at hand. For example, you can develop a specialized list of heuristics for specific audiences, like senior citizens, children, or disabled users, based on a review of the literature.
Participants and Other Stakeholders
The basic heuristic inspection does not involve users of the product under consideration. As originally proposed by Nielsen and Molich (1990), the heuristic review method was intended for use by people with no formal training or expertise in usability. However, Nielsen (1992) and Desurvire, Kondziela, and Atwood (1992) found that usability experts would find more issues than non experts. For some products a combination of usability practitioners and domain experts would be recommended.
The stakeholders are those who will benefit from the cost savings that may be realized from using a “discount” (i.e. low cost) usability methods. These stakeholder may include the ownership and management of the company producing the product and the users who will purchase the product.
Who Can Facilitate
Heuristic evaluations are generally organized by a usability practitioner who introduces the method and the principles, though with some training, other members of a product could facilitate.
Data Analysis Approach
The data are collected in a list of usability problems and issues. Analysis can include assignment of severity codes and recommendations for resolving the usability issues. The problems should be organized in a way that is efficient for the people who will be fixing the problems.
Discuss the usability issues with the product team. Track what problems are fixed, deferred, and viewed as “not a problem” by the product team.
Costs and Scalability
People and Equipment
There is no special equipment required for a heuristic evaluation, other than the computer or other hardware (PDA, cell phone, etc.) used to run the application. The cost will reflect the number of evaluators, their level of usability and domain expertise, and the amount of time they put into the evaluation. As originally proposed by Nielsen and Molich (1990), the heuristic review method was intended for use by people with no formal training or expertise in usability. However, Nielsen (1992) and Desurvire, Kondziela, and Atwood (1992) found that usability experts would find more issues than non experts, and Nielsen (1992) found that double experts (evaluators with usability expertise and expertise in the domain in which the software is used) will find more issues than usability experts. Short training sessions on the list of heuristics may add some costs, but make for more effective evaluations.
Molich and Dumas (2005) reported that expert reviews (which included one heuristic review) conducted for the CUE-4 study took significantly less time than usability tests, and that the expert reviews identified the same number and quality of issues.
None (unless the list of heuristics will be used to evaluate accessibility).
None (unless the list of heuristics will be used to evaluate localization issues).
Ethical and Legal Considerations
For mission-critical products, the heuristic evaluation should not be the sole evaluation method for examining potential usability problems. More formal methods including summative testing and formal user interface inspections may be required to examine subtle interactions between the user, task, and product that would create serious errors.