What automated UX testing can catch

A browser can measure many usability defects on a live page without a person watching: a layout wider than the screen, a button covered by a banner, text that is cut off, a tap target too small to hit, a page that jumps while it loads.

It cannot judge whether the words make sense, whether the navigation matches how people think, or whether the page helps them do what they came for. Those need a person, and often a real user.

Our guide to running a website UX audit shows where automated checks fit among the other steps.

Three things called automated UX testing

The phrase covers three different jobs:

• Scripted functional tests. A script clicks through a flow, such as sign-up or checkout, and checks that each step works. It tells you the flow does not break; it does not tell you the flow is easy.
• Automated research tools. Unmoderated usability tests, session recordings, heatmaps and surveys collect what real people do and say. The collection is automated; the people are real, and someone still has to interpret what they did.
• Measuring the rendered page. A browser loads each page, at several screen sizes, and measures what it can: geometry, colour, focus, timing. This post is about that third kind.

How a browser measures a page

A browser that renders a page knows a great deal about it. Through the interfaces a page’s own scripts use, plus trusted key presses sent by browser automation, a test can read:

• Computed styles: the colour, font size, line height and spacing the browser actually applied, after every stylesheet.
• Geometry: where each element is, how big it is, and whether it fits inside the screen or its container.
• What is on top: which element sits at a given point, so a test can tell whether a control is covered by another element.
• Keyboard behaviour: where focus goes when the test presses Tab, and whether the focused control looks any different.
• Timing: when the largest content appeared, how much the layout moved, and how long the page took to respond to an input.

A good test waits for the page to settle first: fonts loaded, scripts run, lazy content in place. It repeats the checks at more than one screen width, because many defects appear at only one.

What a browser can measure

Each defect below says how a browser measures it, what evidence a finding should carry, and where the measurement stops.

Layout

• Page wider than the screen. How it is measured: the document is wider than the window at a given width. Evidence: the width, the screen width, and the element that sticks out. Limit: content that needs a two-dimensional layout, such as a data table, may scroll sideways in its own container and still pass WCAG’s reflow criterion, as long as the rest of the page reflows; a person decides.
• Control off screen. How it is measured: a link or button sits beyond the edge of the screen. Evidence: the control and its position. Limit: a control meant to slide in, such as a closed menu, or a skip link that appears on focus, is off screen on purpose.
• Control covered by something else. How it is measured: the element at the control’s centre, or at points inside it, is something other than the control or its contents. Evidence: the control, what covers it, and a screenshot. Limit: the test sees the page at the moments it checks, not every scroll position.
• Text covered or clipped. How it is measured: something opaque is painted over the text, or the text runs past the edge of a container that hides overflow. Evidence: the text and the container. Limit: deliberate truncation with an ellipsis is a design choice a person has to judge.
• Lines too long to read. How it is measured: characters per line of body text. Evidence: the measured line length. Limit: a long line is a readability problem, not a failure at WCAG Level AA; the threshold is a choice.

Touch and pointer

Targets too small. How it is measured: the size of each link and button, and the space around it. Evidence: the size, and the neighbours it crowds. Limit: WCAG 2.2’s 2.5.8 also passes a small target with enough space around it, one whose function another full-size control on the page offers, a link in a sentence, a control whose size the browser sets and the author has not changed, and a size that is essential or legally required; a person judges all but the spacing.

Keyboard

• Control Tab cannot reach. How it is measured: an element that acts as a control, such as one with an onclick attribute, has no control role such as button or link and is not in the Tab order, so Tab never reaches it. Evidence: the element’s selector, tag, text, role and tabindex. Limit: a test can miss a control that appears only after an action, or one whose click handler is attached by script.
• Focus not visible. How it is measured: when focus moves to a control, its outline, border, shadow and background do not change. Evidence: the control’s styles before and after. Limit: whether a change is clearly visible, rather than merely present, can need a person, and a focus style drawn only by a text colour, an underline or a pseudo-element can be missed by this comparison, so a person confirms it.
• Focus trapped. How it is measured: Tab returns to a control it already reached while much of the page is still unreached. Evidence: how many of the page’s controls Tab reached, and the stops in order. Limit: a modal dialog is meant to hold focus until it closes, as long as the keyboard can close it.

Reading

• Text contrast too low. How it is measured: the ratio between the text colour and the colour behind it, read from computed styles. Evidence: both colours and the ratio. Limit: text over an image cannot be decided from styles alone.
• Content lost under text spacing. How it is measured: the test applies WCAG’s larger line, paragraph, letter and word spacing and looks for text that is cut off or hidden, or a control that disappears. Evidence: the element before and after. Limit: it checks the spacing values WCAG names, not every user setting.

Performance

Slow loading, layout that jumps, slow responses. How it is measured: the browser reports Largest Contentful Paint, Cumulative Layout Shift and, when the test interacts with the page, Interaction to Next Paint. Evidence: the values, the element that shifted, and the largest element painted. Limit: a lab run is one load on one simulated device, and its Interaction to Next Paint covers only the interactions the test made; Google’s thresholds are meant for the 75th percentile of real visits, which only field data shows.

Links and consent

• Broken links and redirect chains. How it is measured: the test requests each link and reads the response. Evidence: the link and the status code or the chain. Limit: a page that answers “200 OK” with a “not found” message looks fine to a status check.
• Consent banners with no way to decline. How it is measured: the banner has no reject or necessary-only button, or cannot be closed. Evidence: the banner’s buttons. Limit: whether the wording is fair is a judgment.

Most of these are also accessibility checks. Our post on automated and manual accessibility testing goes through the WCAG criteria one by one.

Two defects in code

A fixed width is a common cause of a page wider than the screen:

.promo-banner {  width: 1200px;}

.promo-banner {  box-sizing: border-box;  width: 100%;  max-width: 1200px;}

A fixed bar at the bottom of the screen covers controls near the end of a page unless the page leaves room for it:

.cookie-bar {  position: fixed;  inset: auto 0 0 0;  height: 96px;}

.cookie-bar {  position: fixed;  inset: auto 0 0 0;  height: 96px;}​html {  padding-bottom: 96px;  scroll-padding-bottom: 96px;}

What makes a finding trustworthy

A measured finding is only as useful as its evidence. Look for findings that name:

• the page, the screen width and the element, by selector;
• the measured values, such as a ratio, a width or a timing;
• a screenshot of the moment it was measured;
• the rule that fired, so the same check can run again after the fix.

A tool that reports a number without the element, or a pass without saying what it measured, leaves you unable to check either. The W3C’s own advice on evaluation tools is plain: “Tools cannot check all accessibility aspects automatically”, “Human judgement is required”, and “Sometimes evaluation tools can produce false or misleading results.”

A measurement says what is true on the page. It does not say how much it matters. A covered button on the checkout page and a covered link in the footer are the same defect with very different consequences, so a person still decides severity and order: severity first, then reach. Many measured defects come from the template, so check for them on one page of each template, plus pages whose content differs from the rest of their template, as in template-based UX auditing.

What automation cannot judge

The measurements above say nothing about:

• Words. Whether a label, a heading, a button or an error message is clear, and whether it says the same thing as the page around it.
• Structure. Whether the navigation and the categories match how visitors think about what you offer.
• The task. Whether the page helps someone do what they came for, and whether it is the right task to offer them at all.
• Emphasis. Whether the most important thing on the page looks the most important.
• Trust. Whether the page looks credible, the price is clear and nothing surprises the visitor at the last step.
• Accuracy. Whether the content is right and current.

These need an expert review, and for the key journeys, real users. Our checklist for UX audits marks which checks a tool can measure and which a person has to judge. For the expert review, see heuristic evaluation. For one page that has to convert, use the friction checklist for a landing page audit.