Google-NotebookLM
Avg Crawl Delay (when set) 129s
Mentions
652,912
0.05% of hostnames
Disallow All
97,578
14% of mentions
Allow
550,863
84% of mentions
Disallow All
+ Allow Some
+ Allow Some
129
0% of mentions
Disallow Some
336
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,936
0% of mentions
No Directives
59
0% of mentions
Average Crawl Delay
129
(when set)