On this page

Before changing robots.txt, write the intended result for a particular crawler and path. “Allow the public research notes while leaving the rest of that directory disallowed” is testable. “Improve the AI score” leaves too much undefined.

The Cite Files robots.txt tester accepts pasted text and a path. Its documentation says that the path is compared with the rules rather than fetched. That makes it suitable for checking a proposed file before publishing it, while keeping delivery tests separate.

Start with an explicit fixture

In the fictional ExampleBot fixture, adding Allow for /research/public/ changes the public notes path to allowed while private notes remains disallowed.
Original before-and-after illustration interpreted against RFC 9309. This is a fictional policy exercise; it does not claim that any real crawler visited a page. Awesome Websites; original explanatory graphic · Original site graphic Enlarge image

The following is a fictional exercise for a made-up crawler called ExampleBot. It is not a recommended policy for a real provider. Before the change:

User-agent: ExampleBot
Disallow: /research/

Proposed version:

User-agent: ExampleBot
Disallow: /research/
Allow: /research/public/

The longer matching path decides under RFC 9309, section 2.2.2. For this fixture, the expected results are:

ExampleBot pathBeforeProposed versionReason
/research/public/notesDisallowedAllowedThe new public-path rule is more specific
/research/private/notesDisallowedDisallowedOnly the research-directory rule matches
/aboutAllowedAllowedNeither path rule matches

This table is a worked interpretation of the fixture, not a live crawler observation. Include both the path you want to change and paths whose result should remain the same.

Check the applicable group

RFC 9309 says matching named groups are combined; the wildcard group applies when no named group matches. A named group does not simply inherit the wildcard group's restrictions. See section 2.2.1.

For your real file, record the actual operator-documented token. Inspect all groups naming it and test the complete proposed file. Do not paste the fictional ExampleBot name into a tool that only reports a fixed list of real crawlers and then assume it tested that name.

In the Cite Files tester, paste your real file, enter the exact path, and read the result for the intended listed token. Repeat with the proposed file and each control path. Save the deciding rule as well as the verdict.

Keep a change log

Use this original worksheet to make the change reviewable:

FieldRecord
Policy intentionToken, path and intended result
BeforeFull served file, retrieval time and expected path results
ProposalExact file change and results for all control paths
After deploymentNewly retrieved file and repeated path checks
Delivery checkStatus and body of a separate authorised page request
Remaining uncertaintyCaching, genuine crawler behaviour or unexplained differences

Keep the existing file as a restorable version. After an approved change, fetch the publicly served file again; checking an editor's saved copy does not show what the server delivers. RFC 9309 permits caching, so do not treat an immediate lack of crawler activity as proof that the rule failed.

Investigate delivery independently

A rules tester evaluates crawl policy. The separate Cite Files delivery test describes a request comparison for delivery. If your proposed rule allows a path but a probe receives 403, save the response and inspect the corresponding server or CDN log. The status alone does not identify which component refused it.

Leave protections in place while diagnosing the cause. Finish with two findings: whether the served rules express the intended policy, and what the observed requests received. Neither finding promises a crawl or a citation.