Claude-Web
Avg Crawl Delay (when set) 12s
Mentions
3,284,973
0.25% of hostnames
Disallow All
864,194
26% of mentions
Allow
64,394
1% of mentions
Disallow All
+ Allow Some
+ Allow Some
3,542
0% of mentions
Disallow Some
18,224
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
2,280,611
69% of mentions
No Directives
53,969
1% of mentions
Average Crawl Delay
12
(when set)