Gigabot
Avg Crawl Delay (when set) 131s
Mentions
1,756,997
0.13% of hostnames
Disallow All
1,682,754
95% of mentions
Allow
4,346
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
483
0% of mentions
Disallow Some
3,634
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
647
0% of mentions
No Directives
65,129
3% of mentions
Average Crawl Delay
131
(when set)