Claude-User
Avg Crawl Delay (when set) 14s
Mentions
139,403
0.01% of hostnames
Disallow All
50,850
36% of mentions
Allow
33,449
23% of mentions
Disallow All
+ Allow Some
+ Allow Some
34
0% of mentions
Disallow Some
29,847
21% of mentions
Disallow Some
+ Allow Some
+ Allow Some
24,595
17% of mentions
No Directives
616
0% of mentions
Average Crawl Delay
14
(when set)