Storebot-Google
Avg Crawl Delay (when set) 31s
Mentions
96,809
0.01% of hostnames
Disallow All
12,212
12% of mentions
Allow
32,361
33% of mentions
Disallow All
+ Allow Some
+ Allow Some
1,202
1% of mentions
Disallow Some
3,778
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
41,018
42% of mentions
No Directives
6,229
6% of mentions
Average Crawl Delay
31
(when set)