On this page
A useful access report answers a narrow question: what did this request receive, from this address, at this time? Keep that observation separate from whether an assistant will cite the page.
Cite Files' crawler checks compare browser and crawler-labelled requests. Its methodology describes a single request per crawler from one address, so the result is a diagnostic sample. It is not a visit from the crawler's own infrastructure.
Separate three questions

Use this worksheet for one public page you own or are authorised to test. The rows are separate tests; a tick in one does not fill the others.
| Question | Evidence to save | What that evidence cannot establish |
|---|---|---|
| What does the crawl policy say? | The served robots.txt, crawler token and exact page path | Whether the server delivers the page |
| What did this request receive? | Time, URL, request identity, response status and saved body | What a verified provider crawler receives elsewhere |
| Was the page used in an answer? | The actual answer, query, time and cited URL | Reliable inclusion for other queries or future answers |
Robots.txt expresses rules for cooperating crawlers; it is not access authentication. That distinction comes from RFC 9309, section 3. Keep private information behind appropriate access controls.
Record the body, not just the status
Here is a suggested manual check. Choose one public article and note a distinctive sentence currently visible on it. Save the response headers and body from a normal request. Record redirects and check whether the returned body contains the article, an error, a login screen or a challenge.
For a crawler-labelled comparison, record the exact user-agent string as well. Describe it as a probe using that string. Do not label it a successful visit by the provider's bot. Cloudflare's IP-validation documentation describes checks involving bot-owner IP lists or reverse DNS as well as user-agent patterns. Changing the name in your request does not reproduce those conditions.
Keep the request rate low. A refusal is evidence to investigate in your own request logs, not a reason to remove bot protection or repeat requests aggressively.
Interpret a mismatch carefully
This is a fictional log, not a test of a named website:
| Request | Status | Saved body | Defensible observation |
|---|---|---|---|
| Browser-labelled probe | 200 | Article text | This probe received the article |
| Crawler-labelled probe | 403 | Refusal page | This probe was refused |
| Later verified crawler entry in server logs | Not yet found | Not available | Actual crawler access remains unverified |
The second row does not identify the cause. Compare its request identifier and timestamp with the CDN and application logs before attributing the refusal to a particular rule. A 200 response also needs inspection: a challenge or generic page is not the requested article.
The documented crawler-test limitations explain its structural comparisons and their limits. Keep the raw observations alongside its verdict so another person can check your interpretation.
Measure visibility separately
Google says that its AI search features require ordinary Search eligibility and do not guarantee crawling, indexing or serving even when requirements are met. It also says special AI text files are unnecessary for those features. Those statements concern Google's products, not every assistant. See Google's AI features guidance.
Your finished record should therefore contain an access finding and a separate visibility finding. “The probe received the article; citation has not been observed” is a useful result. It tells you exactly what remains to be measured.
