Claude-User
Avg Crawl Delay (when set) 14s
Mentions
344,451
0.02% of hostnames
Disallow All
177,486
51% of mentions
Allow
67,481
19% of mentions
Disallow All
+ Allow Some
+ Allow Some
1,284
0% of mentions
Disallow Some
49,610
14% of mentions
Disallow Some
+ Allow Some
+ Allow Some
45,418
13% of mentions
No Directives
3,091
0% of mentions
Average Crawl Delay
14
(when set)