Google-NotebookLM
Avg Crawl Delay (when set) 14s
Mentions
467,662
0.03% of hostnames
Disallow All
89,477
19% of mentions
Allow
374,123
79% of mentions
Disallow All
+ Allow Some
+ Allow Some
112
0% of mentions
Disallow Some
221
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,699
0% of mentions
No Directives
21
0% of mentions
Average Crawl Delay
14
(when set)