OpenRobotsTXT

AdIdxBot

Avg Crawl Delay (when set) 9s

Mentions

325,284

0.02% of hostnames

Disallow All

31,630

9% of mentions

Allow

25,166

7% of mentions

Disallow All
+ Allow Some

1,506

0% of mentions

Disallow Some

45,380

13% of mentions

Disallow Some
+ Allow Some

45,541

14% of mentions

No Directives

176,056

54% of mentions

Average Crawl Delay

9

(when set)