SemrushBot
Avg Crawl Delay (when set) 16s
Mentions
6,408,759
0.44% of hostnames
Disallow All
4,750,356
74% of mentions
Allow
134,512
2% of mentions
Disallow All
+ Allow Some
+ Allow Some
3,810
0% of mentions
Disallow Some
81,359
1% of mentions
Disallow Some
+ Allow Some
+ Allow Some
77,429
1% of mentions
No Directives
1,361,058
21% of mentions
Average Crawl Delay
16
(when set)