Glossary
robots.txt
robots.txt is a plain-text file at the root of a website that tells web crawlers which URLs they can and cannot access.
robots.txt is a long-standing web convention. It lives at /robots.txt on the apex domain. It uses a simple syntax: 'User-agent: *' followed by 'Allow:' and 'Disallow:' rules. A 'Sitemap:' URL line lists the site's XML sitemap.
robots.txt is a hint, not a hard wall — well-behaved crawlers (Googlebot, Bingbot, Anthropic's ClaudeBot, OpenAI's GPTBot) respect it; malicious crawlers ignore it. To truly block content from search results, combine robots.txt with the noindex meta tag and authentication.
robots.txt is parsed per-bot. You can set different rules for different user-agents — common pattern is to allow all real search engines and disallow specific AI training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) if you don't want your content training models. The policy decision (allow or block) is yours; both choices are common in 2026.
Beyond Allow/Disallow, modern robots.txt entries reference crawl-delay (slow down a bot), specific path patterns, and the sitemap URL. Some bots (Yandex, Baidu) honor crawl-delay; Google ignores it in favor of its own crawl-budget logic.
Common mistakes: blocking critical paths by accident (Disallow: / blocks the whole site), expecting robots.txt to block bad actors (it doesn't), forgetting to update robots.txt when moving a site to a new platform, and failing to reference the sitemap URL (Google can find it anyway, but the line is best practice).
Example
Disallowing /admin/ and /api/ in robots.txt prevents crawl-budget waste on internal routes that don't belong in search. A 'Disallow: /' on a staging environment prevents Googlebot from indexing the staging site before launch.