OpenRobotsTXT

facebookexternalhit

Avg Crawl Delay (when set) 25s

Mentions

1,243,157

0.11% of hostnames

Disallow All

154,536

12% of mentions

Allow

502,088

40% of mentions

Disallow All
+ Allow Some

23,562

1% of mentions

Disallow Some

58,179

4% of mentions

Disallow Some
+ Allow Some

314,390

25% of mentions

No Directives

190,358

15% of mentions

Average Crawl Delay

25

(when set)