anthropic-ai
Avg Crawl Delay (when set) 31s
Mentions
6,745,319
0.48% of hostnames
Disallow All
2,591,213
38% of mentions
Allow
205,569
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
5,324
0% of mentions
Disallow Some
13,209
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,871,578
57% of mentions
No Directives
58,375
0% of mentions
Average Crawl Delay
31
(when set)