Claude-Web
Avg Crawl Delay (when set) 15s
Mentions
2,236,135
0.15% of hostnames
Disallow All
1,630,527
72% of mentions
Allow
169,695
7% of mentions
Disallow All
+ Allow Some
+ Allow Some
6,571
0% of mentions
Disallow Some
71,185
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
334,600
14% of mentions
No Directives
23,436
1% of mentions
Average Crawl Delay
15
(when set)