6️⃣ Blocking AI Crawlers with robots.txt (2023-07)
Can you prevent your content from being used by #AI companies? Not really. But here are some attempts to get you at least part of the way there. And what we should be doing to have the choice.
#LLM #ChatGPT
https://netfuture.ch/2023/07/blocking-ai-crawlers-robots-txt-chatgpt/
Marcel Waldvogel /
npub1pv…pqjey
2023-12-23 12:25:48
in reply to nevent1q…u9hz
Author Public Key
npub1pvwwfcwuspuwyjmzh5jlh3xc0y32lvyyg8n45tvsq2wm2eskvwysupqjeySeen on
wss://relay.noswhere.comPublished at
2023-12-23 12:25:48Event JSON
{
"id": "3a72888226709d7da27aa84c329edd42ebb79ca024723f1996f764d4cc7d86e7",
"pubkey": "0b1ce4e1dc8078e24b62bd25fbc4d87922afb08441e75a2d90029db566166389",
"created_at": 1703330748,
"kind": 1,
"tags": [
[
"e",
"9882d95c0f0c5d27c707c76cfc8de451000aacd0dec75c6a6ab8b7d5c3526915",
"wss://relay.mostr.pub",
"reply"
],
[
"t",
"ai"
],
[
"t",
"LLM"
],
[
"t",
"chatgpt"
],
[
"proxy",
"https://waldvogel.family/users/marcel/statuses/111629483923150804",
"activitypub"
]
],
"content": "6️⃣ Blocking AI Crawlers with robots.txt (2023-07)\nCan you prevent your content from being used by #AI companies? Not really. But here are some attempts to get you at least part of the way there. And what we should be doing to have the choice.\n\n#LLM #ChatGPT \nhttps://netfuture.ch/2023/07/blocking-ai-crawlers-robots-txt-chatgpt/",
"sig": "5fdbc06411f650fd498f0139f826d5c30decafcbb6a839fadf21f81a05a238cdce1eb75fbbac466213fef874e6d5c2b7b905ec971a826dadb07defa54e1fcfda"
}