Claude-User
Avg Crawl Delay (when set) 14s
Mentions
329,476
0.02% of hostnames
Disallow All
177,245
53% of mentions
Allow
61,832
18% of mentions
Disallow All
+ Allow Some
+ Allow Some
615
0% of mentions
Disallow Some
45,802
13% of mentions
Disallow Some
+ Allow Some
+ Allow Some
40,988
12% of mentions
No Directives
2,960
0% of mentions
Average Crawl Delay
14
(when set)