rogerbot
Avg Crawl Delay (when set) 194s
Mentions
2,391,600
0.17% of hostnames
Disallow All
2,042,028
85% of mentions
Allow
91,156
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
21,295
0% of mentions
Disallow Some
7,469
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
101,250
4% of mentions
No Directives
128,235
5% of mentions
Average Crawl Delay
194
(when set)