Heuristic evaluation for websites: how to run one and what it misses

A heuristic evaluation is an expert review of an interface against a short list of usability principles. A few evaluators go through the site on their own, note each place it breaks a principle, then merge their lists and rate how severe each problem is.

It is cheap and needs no users. It also misses problems: in the study that introduced it, the average evaluator found only between 20 and 51% of the known problems, depending on the interface.

A website UX audit uses it as the expert step, beside automated checks and user testing.

What a heuristic evaluation is

Jakob Nielsen and Rolf Molich described the method at the CHI conference in 1990, in a paper called “Heuristic evaluation of user interfaces”. They defined it as “an informal method of usability analysis where a number of evaluators are presented with an interface design and asked to comment on it.” Their list had nine heuristics. In 1994 Nielsen refined the list, from a factor analysis of 249 usability problems, into the ten heuristics most teams use today.

A heuristic is a rule of thumb, not a requirement. Nielsen Norman Group’s own guide calls them “guidelines, not laws”: breaking one is a reason to look closer, not proof of a problem.

Three methods are often confused with it:

• Usability testing. Real people try real tasks while you watch. A heuristic evaluation predicts problems; a test shows them.
• A UX audit. An audit is broader: it scopes the site, runs automated checks, includes accessibility, ranks the findings and re-checks the fixes. A heuristic evaluation is one of its methods.
• A cognitive walkthrough. Another inspection method, which simulates a user’s problem-solving at each step of a task. It checks at each step whether the user’s goals and what they already know would lead them to the next correct action; a heuristic evaluation instead judges each element it reviews against the list.

How many evaluators you need

Nielsen and Molich ran four experiments. The evaluators were computer science students and readers of a Danish computing magazine, not usability specialists. Each evaluator wrote a report on an interface, and each report was scored against the authors’ list of that interface’s known problems.

• One evaluator misses most problems. On average, a single evaluator found 51%, 38%, 26% and 20% of the known problems in the four interfaces.
• Evaluators find different problems. Weaker evaluators sometimes found hard problems, and stronger ones sometimes missed easy ones, so a group finds more than its best member.
• A few evaluators find much more. From simulated groups built out of the individual reports, the authors expected groups of five to find “about two thirds of the usability problems”, with little gained beyond about ten. They recommended three to five evaluators, working independently.

Later data points the same way. Averaged over six of Nielsen’s own projects, single evaluators found 35% of the problems. Nielsen Norman Group’s current guide still asks for three to five people, who should not see each other’s findings until their own are done.

Read the numbers as a shape, not a forecast. The 1990 interfaces were a videotex service, a design written for the study and two telephone voice-response systems, and the list of known problems was the authors’ own. With only one reviewer, expect to miss much of what a group would find, and lean harder on the other methods below.

The ten heuristics on a website

For each heuristic: what to look for on a website, a failure we made up to show it, and who can find it. A tool can measure only a few parts, an expert judges most of it, and some of it only real users can show.

1. Visibility of system status

Look for: feedback after every action. A button that shows it is working, a basket count that changes, a form that says it was sent, and a sign of where you are in the site and in a multi-step process.

A failure: “Add to basket” changes nothing on screen, so the visitor presses it again and ends up with two.

Who finds it: an expert, by trying each action. WCAG asks that a status message such as “Added to basket” is exposed to assistive technology without moving focus (4.1.3). Checking that usually takes a person with a screen reader. Whether sighted visitors notice the message takes a person too.

2. Match between the system and the real world

Look for: the visitor’s words, not the organisation’s. Menu labels, category names, error text, and units, currencies and date formats for the market you sell to.

A failure: the price list sits under a menu item called “Solutions”, and product pages for consumers show a “SKU” field.

Who finds it: an expert spots the jargon. Only users can tell you which words they look for. No tool measures whether a word means the right thing to your visitors.

3. User control and freedom

Look for: a way out of every state. A close button and the Escape key on dialogs, Back that keeps a visitor’s filters, undo when an item is removed, and a way to change an order before paying.

A failure: a newsletter overlay whose close button sits below the first screen on a phone.

Who finds it: an expert, for most of it. A tool can find the measurable parts: keyboard focus that cannot leave a component (2.1.2), and an overlay that covers the controls under it.

4. Consistency and standards

Look for: the same thing looking and working the same way across templates, and the web’s conventions: the logo goes home, links look like links, and the basket is where visitors expect it.

A failure: the primary button is blue on product pages and green in the checkout, and the header says “Basket” while the checkout says “Cart”.

Who finds it: an expert comparing templates side by side. WCAG makes two parts testable: navigation repeated across pages stays in the same relative order unless the user changes it (3.2.3), and components with the same function are identified consistently (3.2.4). Most tools check one page at a time, so both need a comparison across templates: a person lists the menu order on a page of each template and decides whether two components do the same thing. Visual consistency has its own method, a UI audit against your design system.

5. Error prevention

Look for: constraints and confirmations where a mistake is costly. Fields that state their format, date pickers that block impossible dates, a review step before payment, and a confirmation before anything is deleted.

A failure: “Delete account” sits next to “Download my data”, and one click deletes the account.

Who finds it: an expert, walking each flow. For pages that make a legal commitment or a payment, or change or delete a user’s data, WCAG asks for at least one of these: the submission can be reversed, it is checked for errors with a chance to correct them, or it can be reviewed, corrected and confirmed before it is final (3.3.4). A person checks which of these a flow offers.

6. Recognition rather than recall

Look for: options that stay visible instead of having to be remembered. Labels that stay on screen while you type, filters that show what is applied, and details carried forward through a process.

A failure: the field labels are placeholder text that disappears as soon as you type, and the billing step asks for the address the visitor gave one step earlier.

Who finds it: a tool such as axe-core finds fields with no label at all, which it files under 4.1.2, but it counts a placeholder as a label, so a field whose only label vanishes as you type passes. Whether the labels and instructions are enough (3.3.2) is a question for a person. Making visitors type again what they entered earlier in the same process, when it is neither filled in nor offered for selection, fails Redundant Entry, a WCAG 2.2 Level A criterion (3.3.7), unless one of its three exceptions applies. A person finds it by walking the process.

7. Flexibility and efficiency of use

Look for: shortcuts for frequent visitors that do not get in a new visitor’s way. Search that accepts a product number, saved addresses, reordering, and a quick way past long menus.

A failure: trade customers search by product number every day, and the search returns nothing for one.

Who finds it: an expert, with your analytics or your users to show which shortcuts matter. A tool can find a page with no skip link, landmarks or headings to bypass repeated blocks (2.4.1), and a person confirms that they work.

8. Aesthetic and minimalist design

Look for: each screen leading with what the visitor came for. One primary action, nothing competing with it, and the answer to the visitor’s question near the top.

A failure: on a phone, three promotional strips push the price and the “Add to basket” button below the first screen.

Who finds it: an expert, and users for what they actually look at. A tool measures the defects underneath some clutter, such as a control covered by a banner, a page wider than the screen or lines too long to read: see the usability defects a browser can measure on a live page.

9. Help users recognize, diagnose, and recover from errors

Look for: error messages in plain words, next to the field, that say what is wrong and how to fix it, and keep what the visitor typed. A missing page that helps the visitor go on.

A failure: “Invalid input” at the top of a twelve-field form, with both password fields emptied.

Who finds it: WCAG asks that an error a form detects is identified and described in text (3.3.1), with a suggestion for correcting it where one is known, unless that would put security or the purpose of the content at risk (3.3.3). A tool can enter invalid values and check that a message is tied to the field (1.3.1). Whether the message helps needs a person.

10. Help and documentation

Look for: help where the question comes up. Delivery and returns on the product page, contact details that are easy to find, and short help on any field that needs it.

A failure: the returns policy is a PDF linked from the footer, and the contact link moves from the header to the footer in the checkout.

Who finds it: WCAG 2.2 asks that help repeated across pages, such as contact details or a chat, stays in the same order relative to the rest of the page (3.2.6), which a person checks across templates. Whether the help answers real questions takes users.

How to run one on a website

1. Choose what to review. Pick the templates and the two or three tasks that matter most, such as finding a product and checking out. Review one page that stands for each template; our post on template-based auditing explains how to pick that page.
2. Write the brief. Give every evaluator the same list of heuristics, the tasks, the screen widths (at least a phone width and a desktop width) and a findings sheet. Give evaluators who do not know your field a typical scenario, written out step by step.
3. Evaluate alone. Each evaluator works independently and sees nobody else’s findings until their own are written. Go through each page at least twice: once to get the feel of the flow, once for the detail.
4. Write one problem per finding. Record where it is, at which width, which heuristic it breaks, what happens, and who it affects, with a screenshot.
5. Merge the findings. Put every evaluator’s findings together, combine the duplicates, and discuss the disagreements. Keep the problems only one person found: they are why you used more than one evaluator.
6. Rate severity afterwards. Each evaluator rates every merged problem alone, then you average the ratings. In Nielsen’s experience, ratings from a single evaluator “are too unreliable to be trusted”, while the mean of ratings from three evaluators is satisfactory “for many practical purposes”. His scale is below; because step 7 ranks by reach, leave out how often a problem occurs when you rate, so it is not counted twice.
7. Rank, fix and re-check. Rank the problems by severity, then by reach. Fix each problem at its template, and check the same pages again.

Nielsen’s severity scale weighs how often a problem occurs, how hard it is to overcome and whether it keeps bothering people once they know about it (step 6 leaves out the first):

• 0: not a usability problem at all.
• 1: cosmetic; fix it only if there is time.
• 2: minor; fixing it is a low priority.
• 3: major; important to fix, a high priority.
• 4: a usability catastrophe; fix it before release.

A findings sheet with one row per problem keeps the merge simple. The rows below are examples:

id,template,page,width,heuristic,problem,who it affects,evaluator,severityH-01,Product page,/products/example-chair,390,4 Consistency and standards,"The primary button is green here and blue on every other template",Every visitor,A,H-02,Checkout,/checkout/billing,1280,6 Recognition rather than recall,"Asks again for the address given on the delivery step",Every buyer,B,H-03,Sign-up,/account/new,390,9 Recover from errors,"Shows ""Invalid input"" above the form and empties both password fields",New customers,A,

Leave the severity column empty during the review and fill it in from the averaged ratings.

What a heuristic evaluation misses

• What real users do. Experts predict; users surprise them. Which word visitors search for, how they think your products are grouped, and which problem actually makes them leave show up only when you watch them. Nielsen Norman Group’s guide says it plainly: “heuristic evaluations cannot replace user research.”
• Whether a violation matters. A broken heuristic is not always a problem for your visitors. Severity ratings and testing sort out the false alarms.
• Measurements. An evaluator looking at a page does not measure a contrast ratio, the size of a tap target, the layout at 320 CSS pixels or the loading time on a mid-range phone. A tool does, on a page of each template.
• Assistive technology. A review by eye is not a screen reader test or a full keyboard test. Accessibility needs its own method: see where automated accessibility checks stop and manual testing takes over.
• Solutions and new ideas. The 1990 paper names this itself: the method “sometimes identifies usability problems without providing direct suggestions for how to solve them”, is “biased by the current mindset of the evaluators” and “normally does not generate breakthroughs”.

When to pair it with user testing or automated checks

Each method answers a different question, so the order matters.

1. Automated checks first. Run them on a page of each template, so the evaluators do not spend their time on what a tool can measure. Our checklist of UX checks by page template marks each check as measured or judged.
2. The heuristic evaluation next. It finds the obvious problems before any user sees them, so a usability test is not spent on them.
3. A usability test for the key journeys. A website usability test puts a handful of people who are like your visitors in front of the site, gives each one the tasks from the brief, and watches without helping. Note where they hesitate, what they say and where they give up.
4. The checks again after the fixes. Re-run the automated checks and re-review the changed templates, so a fix that broke something else is caught.