Crawler policy

How ClaimSpyBot reads your site

If you found this page in your server logs, this explains exactly what our crawler is, what it took, and how to slow it down or turn it off.

Last updated 1 September 2026

The short version

ClaimSpyBot reads public marketing pages, obeys robots.txt, requests one page at a time with a delay between requests, and never touches anything behind a login. To block it entirely, disallow ClaimSpyBot in your robots.txt — we re-read it before every crawl and stop within 24 hours.

Identifying the crawler

Every request we make carries this user-agent string:

Mozilla/5.0 (compatible; ClaimSpyBot/1.0; +https://claimspy.io/crawler)

The token to match in robots.txt is ClaimSpyBot. If you see traffic claiming to be ClaimSpy that does not carry this string, it is not us — please tell us and we will help you identify it.

What it collects

ClaimSpyBot is not a search indexer and not a scraper of personal data. On each page it visits it records:

The rendered text of the page, including text that only appears after a drop-down, tab or slider is opened.
Text read out of images by OCR, where a marketing claim is set in a picture rather than in HTML.
A screenshot of the page as rendered.
Standard request metadata — URL, HTTP status, response headers, timing.

From that we extract marketing claims: metrics, positioning statements, comparisons, guarantees and the fine print attached to them. We do not build profiles of individuals, and we do not collect email addresses, phone numbers or form contents for marketing purposes.

What it will not do

It does not log in. No credentials, no gated content, no paywalled pages, no trial signups.
It does not submit forms or click anything that changes state on your server. It only opens things that reveal content already delivered to the browser — drop-downs, tabs, accordions, carousels.
It does not defeat access controls, solve CAPTCHAs to get past a block, or rotate identities to avoid one.
It ignores nothing in robots.txt. Disallow rules matching ClaimSpyBot or * are honoured.

Rate and politeness

Behaviour	What we do
Concurrency	One request at a time per host. We never open parallel connections to the same site.
Delay	At least 2 seconds between requests, and we honour a longer `Crawl-delay` if your `robots.txt` sets one.
Volume	Capped by the customer's plan — between 50 and 500 pages per site, per crawl.
Frequency	Weekly or daily depending on the customer's plan. Never more than once per day.
Errors	We back off on `429` and `5xx` responses, and abandon the crawl after repeated failures rather than retrying hard.
Caching	We send conditional requests where possible so unchanged pages cost you a `304` rather than a full response.

How to block or limit it

Block it entirely

Add this to the robots.txt at the root of your domain:

User-agent: ClaimSpyBot
Disallow: /

We re-read robots.txt at the start of every crawl, so this takes effect within 24 hours at the latest.

Block part of it

Ordinary Disallow paths work as you would expect:

User-agent: ClaimSpyBot
Disallow: /careers/
Disallow: /blog/

Slow it down

User-agent: ClaimSpyBot
Crawl-delay: 10

Ask us directly

If you would rather not edit robots.txt, email hello@claimspy.io from an address at the domain in question and we will add it to our internal block list. We will confirm when it is done.

Why we crawl at all

ClaimSpy customers track the public marketing claims their competitors make, so that they can answer them accurately. This is work their teams already do by hand before every competitive call — reading a rival's pricing page, noting that a statistic changed. We do it consistently, and we keep a dated record so that a claim can be cited as it stood on a given day.

We think that is a legitimate use of publicly published marketing material. We also think you are entitled to opt out of it, which is why blocking us takes one line and we do not make you ask.

Contact

Questions about crawler traffic, or a request to be excluded: hello@claimspy.io. We answer crawler mail before anything else in the inbox.