Lauren Weinstein on Nostr: IMPORTANT: #Google site crawling policy rumor: I've been getting a lot of queries ...
IMPORTANT: #Google site crawling policy rumor: I've been getting a lot of queries over the last few days regarding a recent rumor spreading rapidly that Google has or will soon be changing their site crawling policy to omit indexing of sites that do not have a robots.txt file. This rumor appears to probably trace back to this Google webmaster community video:
https://support.google.com/webmasters/community-video/360202946/fix-robots-txt-unreachable-error-website-not-indexingThis video does seem to say that robots.txt is the first file on a site Google will attempt to retrieve, and if it can't find it the crawling and indexing stops at that point. Not crawling/indexing sites without robots.txt files would make no sense to me -- vast numbers of older sites and others have never had robots.txt files because they never felt a need to publish crawling restrictions.
Also note that this Google support doc:
https://support.google.com/webmasters/answer/6062598?hl=enexplicitly says that robots.txt files are not required for crawling/indexing.
Several sources at Google (thanks all!) have now confirmed directly to me that robots.txt files are NOT required for indexing and no change to this policy is in the pipeline. Of course, if you have a sitemap that specifies a robots.txt file that doesn't actually exist you might create a suboptimal crawling situation, but again Google's official statement to me on this is that robots.txt files are NOT required to be present for a site to be crawled and indexed by Google.
Published at
2026-01-15 17:48:33 GMTEvent JSON
{
"id": "5a5eb14910895f14368491a7371005b5242b80bc38f8bee5fea37cbd05b86337",
"pubkey": "f8347ec37d9f23959ba62dd8391fbcade87a7c071205c96f890432e5c047a6ed",
"created_at": 1768499313,
"kind": 1,
"tags": [
[
"t",
"google"
],
[
"proxy",
"https://mastodon.laurenweinstein.org/@lauren/115900371028558067",
"web"
],
[
"imeta",
"url https://mastodon.laurenweinstein.org/system/media_attachments/files/115/900/370/720/781/109/original/da9d7fb8c646efc9.jpg",
"m image/jpeg"
],
[
"proxy",
"https://mastodon.laurenweinstein.org/users/lauren/statuses/115900371028558067",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.laurenweinstein.org/users/lauren/statuses/115900371028558067",
"pink.momostr"
],
[
"-"
]
],
"content": "IMPORTANT: #Google site crawling policy rumor: I've been getting a lot of queries over the last few days regarding a recent rumor spreading rapidly that Google has or will soon be changing their site crawling policy to omit indexing of sites that do not have a robots.txt file. This rumor appears to probably trace back to this Google webmaster community video:\n\nhttps://support.google.com/webmasters/community-video/360202946/fix-robots-txt-unreachable-error-website-not-indexing\n\nThis video does seem to say that robots.txt is the first file on a site Google will attempt to retrieve, and if it can't find it the crawling and indexing stops at that point. Not crawling/indexing sites without robots.txt files would make no sense to me -- vast numbers of older sites and others have never had robots.txt files because they never felt a need to publish crawling restrictions.\n\nAlso note that this Google support doc:\n\nhttps://support.google.com/webmasters/answer/6062598?hl=en\n\nexplicitly says that robots.txt files are not required for crawling/indexing.\n\nSeveral sources at Google (thanks all!) have now confirmed directly to me that robots.txt files are NOT required for indexing and no change to this policy is in the pipeline. Of course, if you have a sitemap that specifies a robots.txt file that doesn't actually exist you might create a suboptimal crawling situation, but again Google's official statement to me on this is that robots.txt files are NOT required to be present for a site to be crawled and indexed by Google.\nhttps://mastodon.laurenweinstein.org/system/media_attachments/files/115/900/370/720/781/109/original/da9d7fb8c646efc9.jpg\n",
"sig": "90d1df6aa18cba72e0e5d668d61c39e1c53d76577fd32c9a88fd3152fe9d852d594beb2d61786e92b2601036a318097793bbf9007edcd1711e96f0898b90385a"
}
