anthropic-ai
Avg Crawl Delay (when set) 16s
Mentions
5,294,666
0.45% of hostnames
Disallow All
2,014,832
38% of mentions
Allow
58,357
1% of mentions
Disallow All
+ Allow Some
+ Allow Some
3,689
0% of mentions
Disallow Some
8,555
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,155,530
59% of mentions
No Directives
53,673
1% of mentions
Average Crawl Delay
16
(when set)