YourUXAgentCrawler

Your UX Agent uses a bounded crawler to inventory publicly available pages and internal links before selecting representative pages for a UX and accessibility audit.

How the crawler behaves

The crawler identifies itself as YourUXAgentCrawler, requests one document at a time, waits at least one second between requests, and follows applicable robots.txt directives and supported crawl delays. It stays on the requested site, apart from an apex-to-www canonical redirect, and does not submit forms or follow unsafe action URLs.

What it collects

Collection is limited to unauthenticated public HTTP and HTTPS responses. We retain a normalized page inventory, response facts, structural digests, and internal-link edges. We do not use customer credentials, cookies, browser storage, or retain raw page HTML.

Safety limits and coverage

A run is capped at 2,000 HTML pages or 45 minutes and respects response-size, redirect, retry, and throttling limits. Sitemap coverage is a comparison with pages reached through internal links; it cannot prove that every URL on a site is known. JavaScript-only links may not be discovered by the HTTP crawler.

How to control it

The crawler reads the robots.txt group for YourUXAgentCrawler, or the * group when there is none, and follows its Disallow, Allow and Crawl-delay lines. To keep it off your whole site, add User-agent: YourUXAgentCrawler followed by Disallow: /. If your robots.txt cannot be read, other than a 404 or 410 response, the crawl stops rather than guess.

Audits use the same name

Audits and free scans load pages in a real Chromium browser whose user agent also contains YourUXAgentCrawler/1.0 and a link to this page. They are not the crawler, and they behave differently. They do not read robots.txt: they open the pages a customer's project lists or chooses to represent each template, or the one page a visitor submits for a free scan. They run the page's scripts, render it at several screen sizes, and check its internal links with HEAD and GET requests.

When a customer adds a test account for their site, the audit signs in with it and audits the signed-in pages the customer lists. When a customer turns on the form-error check, it types invalid values into form fields to see how errors are shown; it never submits a form, and it blocks every request other than GET while it does. A full audit can also follow a journey, clicking and typing as a visitor would: one the customer sets up, or one we pick from a fixed list, such as following the main call to action or finding how to contact the business.

Blocking requests whose user agent contains YourUXAgentCrawler stops both the crawler and audits. Our servers also make a few direct requests under other names: Your UX Agent sitemap discovery/1.0 fetches a site's sitemap and similar files when a customer sets up a project or starts an audit, and UX-Agent transport preflight checks the connection before an audit.

Allow YourUXAgentCrawler in your bot protection

If bot protection or a firewall in front of your site blocks us, the crawler, a free scan or an audit cannot open the pages it blocks, so we cannot check them. To let us through, add a rule that allows requests whose user agent contains YourUXAgentCrawler. Look in your bot protection or firewall settings for a rule that allows requests, or skips its checks, by user agent. Some checks cannot be skipped this way; your provider's documentation says which.

We do not publish a list of IP addresses, so match on the user agent, not on an address. Anyone can send that user agent, so the rule lets their requests through too. If that matters to you, limit the rule to the parts of your site you want audited, including the scripts, styles and images those pages load.