How a UX heuristic analysis is made

What is examined

The public pages a visitor reaches without signing in: the home page and one of each kind behind it, such as search, a product, a category, help or settings. A page built like one already examined is counted, not read again. Each is read at desktop (1440 pixels wide) and phone sizes, usually tablet too; every analysis lists its sizes.

Most sites are read by the 4th Culture engine, which opens each page in a browser at each size, saves what it saw and runs the checks below. A site that refuses an automated visitor is read by hand in an ordinary browser, and its analysis says so in its method line.

What a finding is

Every finding says what was seen, the observation; why it matters to the people using the page, the implication; and what to do, the recommendation, with the screenshot it is about and the measurements behind it. Each cites one heuristic, and a failure of WCAG 2.2 names the success criterion. The heuristics:

• Visibility of system status
• Match with the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency
• Aesthetic and minimalist design
• Recovering from errors
• Help and documentation
• Mobile and touch
• Forms and input
• Accessibility
• Content and tone
• Navigation and wayfinding
• Visual hierarchy
• Trust and credibility
• Performance as felt
• Conversion

The four levels

Every finding has one of four levels, judged against this rubric:

High

A group of users cannot complete a primary task, or a WCAG 2.2 level A failure blocks a primary task for people using assistive technology.

Medium

A primary task can be completed only with difficulty or with risk of a wrong outcome (wrong price, wrong item, lost input), or any WCAG 2.2 A or AA failure on a primary task path that does not block it.

Low

Difficulty on secondary pages or paths, a WCAG AA failure away from the primary paths, or an inconsistency that costs time but not the task.

Best practice

No harm we could measure; the site departs from a convention worth following.

Primary task pages (by role): home, search, category, product, cart, checkout, sign-in; for other sectors their equivalents (plans, booking, quote, account opening, find a doctor), mapped in each report’s roles.

The page roles counted as primary: home, search, category, product, cart, checkout, sign-in, data and privacy controls. Data and privacy controls count where a site exists to give them, as Google’s search-data controls do.

Counts come from the browser. Levels come from the rubric above. The automated checks count what a browser finds; a finding’s level is judged against the rubric above. The two are tied by one rule: every axe rule of impact serious or critical that fails on a primary-task page is either a finding at medium or above that cites the rule’s WCAG criterion or names the rule, or an exemption with its reason written down.

With this rubric high is rare by design, so the ranking is led by WCAG A and AA failures on primary task paths: these analyses are an accessibility-led benchmark with UX commentary.

A level is not a score. Nothing here rates a site overall or ranks one against another.

What the checks count

Across every page and size examined, a browser counts:

• Accessibility rules failed (axe)
• Tap targets under 24 pixels
• Links that open a new tab
• Console errors on load

The accessibility count comes from axe, an open-source rules engine, run against WCAG 2.2 levels A and AA. It counts distinct rules failed, by axe’s impact (critical, serious, moderate, minor), not elements: one missing label on every product tile is one rule. Each analysis lists its rules with the criterion, the elements and the pages each failed, the only place an element count appears. axe catches the failures a machine can detect. The rest of each analysis is read by a person. A site read by hand has its other counts taken by hand, no axe output, and an analysis that says the rules were not measured.

The vantage point

Every analysis is read from Toronto, Canada, signed out, on the date it gives. Some sites show a Canadian visitor something different: a country banner, another store’s stock, another currency, a page that cannot ship. Where that changes a finding, the finding says so. Nothing that needs an account, an address or a purchase was examined.

What an analysis is not

It observes no real people and replaces no research with them. It is not a full accessibility conformance audit: it covers the pages and states examined, not every page, state or assistive technology. Sites change; each analysis describes the pages on the day they were read.

Each analysis is published for criticism and review, written for the company’s own digital product team. Screenshots are cropped to what a finding is about; they and the trademarks belong to their owners.