AhrefsBot
Avg Crawl Delay (when set) 11s
Mentions
15,811,491
1.37% of hostnames
Disallow All
3,335,534
21% of mentions
Allow
155,140
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,174
0% of mentions
Disallow Some
3,898,500
24% of mentions
Disallow Some
+ Allow Some
+ Allow Some
23,002
0% of mentions
No Directives
8,379,063
52% of mentions
Average Crawl Delay
11
(when set)