Robots.txt
Robots.txt is a plain text file at the top of a website, at yoursite.com/robots.txt, that tells crawlers which parts of the site they may visit. Its rules became an IETF standard, RFC 9309, in September 2022. The standard says crawlers are asked to honor the rules, and that the file doesn't lock anyone out.
Source: IETF, RFC 9309: Robots Exclusion Protocol, checked October 29, 2026
What Robots.txt means for your business
Google says the file is mainly there to keep crawlers from overloading your site. It controls visits, not whether a page shows up in results, so a page you want gone needs noindex or a password. Each host has its own file. Google's spec says a robots.txt on yoursite.com doesn't cover blog.yoursite.com, and paths are case-sensitive, so /Menu and /menu are two different rules.
It's also where you allow or block AI crawlers, one name at a time. Each AI company publishes the names its bots use, and our GPTBot and ClaudeBot entries cover which of their bots collect training data and which help with search. Our Googlebot and Google-Extended entries cover how Google splits Search from Gemini.
We'd treat AI training and AI search as two questions, then write one rule per bot. A single User-agent: * group with Disallow: / blocks every bot that follows the standard, search engines included. The RFC also warns that listing a private folder in the file makes its address public, so a password does that job better. Google's guide to AI Overviews and AI Mode says crawling has to be allowed by your CDN and host too, so we'd read both. If you can't tell which search and AI bots your site turns away, checking your robots.txt and firewall rules together is a check we'd run first.
Last reviewed October 29, 2026. All terms
How AI answers it today
On October 29, 2026, Google's AI Overview called the file a guideline, not a lock, and named GPTBot as an example, citing SEO tool blogs, a site builder's help page and YouTube, not Google or the standard. ChatGPT cited nothing and skipped AI crawlers.
Our run through ChatGPT and Google's AI Overview, October 29, 2026. A sample, not what every buyer sees.
Related terms in crawlers and technical access
Find out where your brand stands.
Tell us what you're trying to figure out. We'll get back to you within one business day with a time to talk.