Claude-User
Avg Crawl Delay (when set) 16s
Mentions
243,341
0.02% of hostnames
Disallow All
144,995
59% of mentions
Allow
43,037
17% of mentions
Disallow All
+ Allow Some
+ Allow Some
181
0% of mentions
Disallow Some
26,127
10% of mentions
Disallow Some
+ Allow Some
+ Allow Some
27,159
11% of mentions
No Directives
1,816
0% of mentions
Average Crawl Delay
16
(when set)