Avg Crawl Delay (when set) 9s
Mentions
223,347
0.02% of hostnames
Disallow All
7,529
3% of mentions
Allow
65,783
29% of mentions
Disallow All
+ Allow Some
+ Allow Some
6,193
2% of mentions
Disallow Some
37,701
16% of mentions
Disallow Some
+ Allow Some
+ Allow Some
57,022
25% of mentions
No Directives
49,112
21% of mentions
Average Crawl Delay
9
(when set)