Common Crawl (CCBot)
Common Crawl is a nonprofit that crawls the web and gives away copies of what it collects, free for researchers, companies and anyone else to download and study. Its crawler is called CCBot, and it gives that name when it visits a site. Common Crawl says it wants web data that anyone can get to and analyze.
Source: Common Crawl, CCBot, checked October 29, 2026
What Common Crawl (CCBot) means for your business
Common Crawl's data doesn't stay with Common Crawl. OpenAI's 2020 research paper says GPT-3 was trained partly on a filtered version of it. Because the archive is public, a page CCBot saves can end up in datasets that many different groups build and train on, long after the visit.
Blocking it is simple. Common Crawl says to add a robots.txt rule that names CCBot. Its FAQ says it keeps checking whether your robots.txt has changed, and that it saves a sample of the web, so not every page on your site will be in it. It also keeps an Opt-Out Ledger, a public list of the legal opt-out requests it has received, and takes requests at info@commoncrawl.org. Its pages don't describe a way to remove pages it already saved. Common Crawl warns that other crawlers pretend to be CCBot, and publishes the real one's IP ranges.
Blocking CCBot is mainly a choice about AI training, since its archive has gone into training sets like GPT-3's. Whether to block it is up to you. For a local business that wants AI tools to know its name and services, we'd lean toward letting it in. If you'd rather your writing stay out of training sets, a CCBot rule in robots.txt says so.
If you'd like a say in which AI training sets your pages end up in, setting your robots.txt rules on purpose is a short job we'd do with you.
Last reviewed October 29, 2026. All terms
How AI answers it today
On October 29, 2026, Google's AI Overview cited Common Crawl's site, Wikipedia, LinkedIn and developer blogs, and called it the main training source for several named AI models. ChatGPT gave a plain definition, said AI companies train on it, and linked Common Crawl's site.
Our run through ChatGPT and Google's AI Overview, October 29, 2026. A sample, not what every buyer sees.
Related terms in crawlers and technical access
Find out where your brand stands.
Tell us what you're trying to figure out. We'll get back to you within one business day with a time to talk.