Claude-User
Avg Crawl Delay (when set) 13s
Mentions
421,985
0.03% of hostnames
Disallow All
275,979
65% of mentions
Allow
60,984
14% of mentions
Disallow All
+ Allow Some
+ Allow Some
520
0% of mentions
Disallow Some
45,247
10% of mentions
Disallow Some
+ Allow Some
+ Allow Some
35,872
8% of mentions
No Directives
3,349
0% of mentions
Average Crawl Delay
13
(when set)