AdsBot
Avg Crawl Delay (when set) 14s
Mentions
611,407
0.04% of hostnames
Disallow All
187,218
30% of mentions
Allow
374,044
61% of mentions
Disallow All
+ Allow Some
+ Allow Some
271
0% of mentions
Disallow Some
1,007
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
35,038
5% of mentions
No Directives
13,825
2% of mentions
Average Crawl Delay
14
(when set)