Avg Crawl Delay (when set) 852s
Mentions
240,452
0.02% of hostnames
Disallow All
31,713
13% of mentions
Allow
208,269
86% of mentions
Disallow All
+ Allow Some
+ Allow Some
8
0% of mentions
Disallow Some
72
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
53
0% of mentions
No Directives
336
0% of mentions
Average Crawl Delay
852
(when set)