OpenRobotsTXT

facebookexternalhit

Avg Crawl Delay (when set) 24s

Mentions

1,489,051

0.11% of hostnames

Disallow All

189,191

12% of mentions

Allow

671,169

45% of mentions

Disallow All
+ Allow Some

25,101

1% of mentions

Disallow Some

63,454

4% of mentions

Disallow Some
+ Allow Some

336,217

22% of mentions

No Directives

203,177

13% of mentions

Average Crawl Delay

24

(when set)