omgilibot
Avg Crawl Delay (when set) 12s
Mentions
2,738,972
0.21% of hostnames
Disallow All
868,239
31% of mentions
Allow
7,285
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,057
0% of mentions
Disallow Some
1,873
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,785,020
65% of mentions
No Directives
53,464
1% of mentions
Average Crawl Delay
12
(when set)