On this page

A useful access report answers a narrow question: what did this request receive, from this address, at this time? Keep that observation separate from whether an assistant will cite the page.

Cite Files' crawler checks compare browser and crawler-labelled requests. Its methodology describes a single request per crawler from one address, so the result is a diagnostic sample. It is not a visit from the crawler's own infrastructure.

Separate three questions

Keep crawl policy, probe delivery and observed citation as three separate findings, each with its own evidence.
Original diagnostic worksheet. A request using a crawler name is a probe, not proof of a genuine provider-crawler visit. Awesome Websites; original explanatory graphic · Original site graphic Enlarge image

Use this worksheet for one public page you own or are authorised to test. The rows are separate tests; a tick in one does not fill the others.

QuestionEvidence to saveWhat that evidence cannot establish
What does the crawl policy say?The served robots.txt, crawler token and exact page pathWhether the server delivers the page
What did this request receive?Time, URL, request identity, response status and saved bodyWhat a verified provider crawler receives elsewhere
Was the page used in an answer?The actual answer, query, time and cited URLReliable inclusion for other queries or future answers

Robots.txt expresses rules for cooperating crawlers; it is not access authentication. That distinction comes from RFC 9309, section 3. Keep private information behind appropriate access controls.

Record the body, not just the status

Here is a suggested manual check. Choose one public article and note a distinctive sentence currently visible on it. Save the response headers and body from a normal request. Record redirects and check whether the returned body contains the article, an error, a login screen or a challenge.

For a crawler-labelled comparison, record the exact user-agent string as well. Describe it as a probe using that string. Do not label it a successful visit by the provider's bot. Cloudflare's IP-validation documentation describes checks involving bot-owner IP lists or reverse DNS as well as user-agent patterns. Changing the name in your request does not reproduce those conditions.

Keep the request rate low. A refusal is evidence to investigate in your own request logs, not a reason to remove bot protection or repeat requests aggressively.

Interpret a mismatch carefully

This is a fictional log, not a test of a named website:

RequestStatusSaved bodyDefensible observation
Browser-labelled probe200Article textThis probe received the article
Crawler-labelled probe403Refusal pageThis probe was refused
Later verified crawler entry in server logsNot yet foundNot availableActual crawler access remains unverified

The second row does not identify the cause. Compare its request identifier and timestamp with the CDN and application logs before attributing the refusal to a particular rule. A 200 response also needs inspection: a challenge or generic page is not the requested article.

The documented crawler-test limitations explain its structural comparisons and their limits. Keep the raw observations alongside its verdict so another person can check your interpretation.

Measure visibility separately

Google says that its AI search features require ordinary Search eligibility and do not guarantee crawling, indexing or serving even when requirements are met. It also says special AI text files are unnecessary for those features. Those statements concern Google's products, not every assistant. See Google's AI features guidance.

Your finished record should therefore contain an access finding and a separate visibility finding. “The probe received the article; citation has not been observed” is a useful result. It tells you exactly what remains to be measured.