AhrefsBot
Avg Crawl Delay (when set) 11s
Mentions
16,372,953
1.17% of hostnames
Disallow All
4,326,478
26% of mentions
Allow
281,256
1% of mentions
Disallow All
+ Allow Some
+ Allow Some
21,391
0% of mentions
Disallow Some
4,061,408
24% of mentions
Disallow Some
+ Allow Some
+ Allow Some
78,362
0% of mentions
No Directives
7,603,948
46% of mentions
Average Crawl Delay
11
(when set)