Web crawler
A web crawler is a program that visits websites on its own and downloads their pages. Google's crawler overview calls crawler a general term for any program that automatically discovers and scans websites, also known as a robot or spider. Google runs common crawlers like Googlebot for its products, and says they always follow robots.txt rules for automatic crawls.
Source: Google Crawling Infrastructure, Overview of Google crawlers and fetchers, checked October 29, 2026
What Web crawler means for your business
Google separates crawlers, which work through sites on their own, from fetchers, which make a single request because a person asked. AI companies draw a similar line between bots that crawl on their own and visits a person sets off, which our AI crawler entry covers.
A crawler can only read what it can reach. A page behind a login, a page no other page links to, or a page your firewall turns away may never be visited. We'd check those three before anything else on a site that seems to be missing from search or AI answers.
Your server logs record each crawler visit with the name it gives, so they show which crawlers come, how often and which pages they ask for. We'd read them before changing any rule. If you'd like to know which crawlers reach your site today and which ones get turned away, a crawl and log check is where we'd begin.
Last reviewed October 29, 2026. All terms
How AI answers it today
On October 29, 2026, Google's AI Overview took its definition from Cloudflare and Wikipedia, also citing crawler vendors, Reddit and YouTube, and listed AI training as one use. ChatGPT gave a similar step-by-step answer naming Googlebot, Bingbot, GPTBot and Amazonbot, and cited nothing.
Our run through ChatGPT and Google's AI Overview, October 29, 2026. A sample, not what every buyer sees.
Related terms in crawlers and technical access
Find out where your brand stands.
Tell us what you're trying to figure out. We'll get back to you within one business day with a time to talk.