Bytespider
Avg Crawl Delay (when set) 64s
Mentions
11,201,158
0.8% of hostnames
Disallow All
8,165,438
72% of mentions
Allow
134,557
1% of mentions
Disallow All
+ Allow Some
+ Allow Some
83,203
0% of mentions
Disallow Some
18,850
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
2,673,359
23% of mentions
No Directives
125,694
1% of mentions
Average Crawl Delay
64
(when set)