Claude-Web
Avg Crawl Delay (when set) 10s
Mentions
4,079,487
0.34% of hostnames
Disallow All
806,566
19% of mentions
Allow
53,600
1% of mentions
Disallow All
+ Allow Some
+ Allow Some
3,231
0% of mentions
Disallow Some
7,933
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,156,031
77% of mentions
No Directives
52,094
1% of mentions
Average Crawl Delay
10
(when set)