User agent
A user agent is any program that asks a website for a page, like a web browser, a search engine's crawler or an app. The web's HTTP standard defines it that way and lists web-crawling robots among its examples. A request usually carries a User-Agent line, a short text label that tells the website what kind of program is asking.
Source: IETF, RFC 9110: HTTP Semantics (sections 3.5 and 10.1.5), checked October 29, 2026
What User agent means for your business
For a business site, the user agent matters most in robots.txt. Each group of rules starts with a user-agent line naming the crawler it's for, like Googlebot or GPTBot. Google says its crawlers follow the group with the most specific name that matches them, and that a group for one crawler isn't combined with the * group meant for everyone. So a Googlebot section replaces the general rules for Googlebot rather than adding to them.
The name in robots.txt is a short token. The label in your server logs is longer. OpenAI's sample for GPTBot starts with Mozilla/5.0 and has GPTBot and a version number near the end, and OpenAI says the version can change. We'd match on the token, not the whole label.
The label is only text, so some bots borrow another's. We'd confirm a visit against the company's published IP addresses before allowing or blocking it. Web Bot Auth, a newer method, has bots sign each request instead. If you're not sure your robots.txt rules say what you meant for each crawler, a line-by-line check of which bots your site lets in is where we'd start.
Last reviewed October 29, 2026. All terms
How AI answers it today
On October 29, 2026, Google's AI Overview defined it as software acting for a person, citing MDN, Wikipedia and security and scraping tool blogs, and mentioned Googlebot and robots.txt. ChatGPT focused on browsers, named spotting bots as one use, and cited nothing.
Our run through ChatGPT and Google's AI Overview, October 29, 2026. A sample, not what every buyer sees.
Related terms in crawlers and technical access
Find out where your brand stands.
Tell us what you're trying to figure out. We'll get back to you within one business day with a time to talk.