Write rules line by line — user-agent, disallow, allow, sitemap, crawl-delay, blank line for a new group — and get a canonical robots.txt. Or paste an existing one and lint it: typos ("disalow"), rules that precede any User-agent, non-absolute sitemap URLs.
Per-bot groups are where it earns its keep: allow everyone but block /admin, while a specific crawler (an AI bot, say) gets shown the door entirely.
Frequently asked questions
Does robots.txt block access?
No — it is a request, not a wall. Honest crawlers obey it; anything else can ignore it. Never use it to hide pages; keep those behind auth.
Where does the file live?
At the domain root (https://example.com/robots.txt), lowercase, one file per host — subdomains each need their own.
中文说明
逐行书写规则——user-agent、disallow、allow、sitemap、crawl-delay，空行分组——生成规范的 robots.txt。也可以粘贴现有文件做检查：拼写错误（"disalow"）、出现在任何 User-agent 之前的规则、非绝对地址的 sitemap。
按爬虫分组是它的价值所在：放行所有人但封住 /admin，同时让特定爬虫（比如某个 AI 机器人）吃闭门羹。
常见问题
robots.txt 能拦截访问吗？
不能——它是请求不是墙。守规矩的爬虫会遵守，其他的可以无视。绝不要用它隐藏页面；该藏的内容放鉴权后面。
文件放在哪里？
域名根目录（https://example.com/robots.txt），全小写，每个主机一份——子域名需要各自独立的文件。