NerdyBot
Avg Crawl Delay (when set) 45s
Mentions
1,415,996
0.1% of hostnames
Disallow All
1,414,885
99% of mentions
Allow
134
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
41
0% of mentions
Disallow Some
82
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
223
0% of mentions
No Directives
631
0% of mentions
Average Crawl Delay
45
(when set)