On this page
Before changing robots.txt, write the intended result for a particular crawler and path. “Allow the public research notes while leaving the rest of that directory disallowed” is testable. “Improve the AI score” leaves too much undefined.
The Cite Files robots.txt tester accepts pasted text and a path. Its documentation says that the path is compared with the rules rather than fetched. That makes it suitable for checking a proposed file before publishing it, while keeping delivery tests separate.
Start with an explicit fixture

The following is a fictional exercise for a made-up crawler called ExampleBot. It is not a recommended policy for a real provider. Before the change:
User-agent: ExampleBot
Disallow: /research/
Proposed version:
User-agent: ExampleBot
Disallow: /research/
Allow: /research/public/
The longer matching path decides under RFC 9309, section 2.2.2. For this fixture, the expected results are:
| ExampleBot path | Before | Proposed version | Reason |
|---|---|---|---|
| /research/public/notes | Disallowed | Allowed | The new public-path rule is more specific |
| /research/private/notes | Disallowed | Disallowed | Only the research-directory rule matches |
| /about | Allowed | Allowed | Neither path rule matches |
This table is a worked interpretation of the fixture, not a live crawler observation. Include both the path you want to change and paths whose result should remain the same.
Check the applicable group
RFC 9309 says matching named groups are combined; the wildcard group applies when no named group matches. A named group does not simply inherit the wildcard group's restrictions. See section 2.2.1.
For your real file, record the actual operator-documented token. Inspect all groups naming it and test the complete proposed file. Do not paste the fictional ExampleBot name into a tool that only reports a fixed list of real crawlers and then assume it tested that name.
In the Cite Files tester, paste your real file, enter the exact path, and read the result for the intended listed token. Repeat with the proposed file and each control path. Save the deciding rule as well as the verdict.
Keep a change log
Use this original worksheet to make the change reviewable:
| Field | Record |
|---|---|
| Policy intention | Token, path and intended result |
| Before | Full served file, retrieval time and expected path results |
| Proposal | Exact file change and results for all control paths |
| After deployment | Newly retrieved file and repeated path checks |
| Delivery check | Status and body of a separate authorised page request |
| Remaining uncertainty | Caching, genuine crawler behaviour or unexplained differences |
Keep the existing file as a restorable version. After an approved change, fetch the publicly served file again; checking an editor's saved copy does not show what the server delivers. RFC 9309 permits caching, so do not treat an immediate lack of crawler activity as proof that the rule failed.
Investigate delivery independently
A rules tester evaluates crawl policy. The separate Cite Files delivery test describes a request comparison for delivery. If your proposed rule allows a path but a probe receives 403, save the response and inspect the corresponding server or CDN log. The status alone does not identify which component refused it.
Leave protections in place while diagnosing the cause. Finish with two findings: whether the served rules express the intended policy, and what the observed requests received. Neither finding promises a crawl or a citation.
