Skip to content
Cloudflare Docs

Enforce robots.txt

AI Audit allows you to enforce robots.txt which instructs bots which webpages they can and cannot access.

To enforce robots.txt:

  1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
  2. Go to AI Audit.
  3. From the dropdown at the top of the page, select a specific subdomain where you wish you enforce robots.txt.
  4. From Summary, select Enforce robots.txt policy.
  5. From the Enforce your robots.txt policy page, select Go to WAF custom rules.
  6. From the New custom rule page, name your custom rule.
    • The page will automatically populate the values for the custom rule.
  7. From Then take action...:
    • For Choose action, select Block.
    • For With response type, select Default Cloudflare WAF block page.
  8. From Place at:
    • For Select order, select Last.
  9. Select Deploy.

This custom rule ensures that bots cannot access the pages specified in your robots.txt file.

For more information, refer to the following resources.