Claude-User
Avg Crawl Delay (when set) 14s
Mentions
149,970
0.01% of hostnames
Disallow All
60,007
40% of mentions
Allow
40,701
27% of mentions
Disallow All
+ Allow Some
+ Allow Some
100
0% of mentions
Disallow Some
21,741
14% of mentions
Disallow Some
+ Allow Some
+ Allow Some
26,347
17% of mentions
No Directives
1,055
0% of mentions
Average Crawl Delay
14
(when set)