SemrushBot
Avg Crawl Delay (when set) 16s
Mentions
6,427,734
0.44% of hostnames
Disallow All
4,767,431
74% of mentions
Allow
135,915
2% of mentions
Disallow All
+ Allow Some
+ Allow Some
3,836
0% of mentions
Disallow Some
81,276
1% of mentions
Disallow Some
+ Allow Some
+ Allow Some
77,453
1% of mentions
No Directives
1,361,588
21% of mentions
Average Crawl Delay
16
(when set)