The Graph · Glossary
AI crawlers and usage controls
GPTBot / ClaudeBot / Google-Extended
GPTBot and ClaudeBot are crawlers associated with collection for model development. Google-Extended is a robots.txt control token without its own HTTP User-Agent. These functions differ from search and user-requested access.
SFT perspective
Document crawler access and permitted uses separately. The individual GPTBot, ClaudeBot and Google-Extended entries explain these distinctions.
Relevant pillars : Structure
Find this term in the complete glossary ↗