Salastil (npub1rys…5m9t)
> I'm getting my Nitter instanced scraped by a botnet that appears to be 100k IP large, they get fed in as fast as I ban them, but I don't believe they assign more than 100 IP to scraping at a time as to not DDOS the site,
WEIRD
> one IP never doing a scrape under 7 seconds so rate limiting wont nab them.
Out of curiosity, what UAs are they using? Tried SSL fingerprinting? You know why they'd be hitting your server, like did you check if DiscordBot or something is in your referrers, or someone linked to it from somewhere, or...?
> something I read about 10+ years ago, a sticky trap. I want to ensnare the bot into a perpetually open http request so that it never completes its loop,
Ah, okay, so you can do this pretty easily with nginx: you can forward to different backends conditionally with one of the (really badly documented) `if` directives. Set up a little script, listen/accept on one end, and then make a connection on the other to the actual upstream. So if your Nitter instance is running on localhost:4444, you have this script listen on localhost:4445. Have it relay all of the traffic upstream and then get the entire response (to avoid jamming up the real server), but trickle the response a few bytes at a time. Some clients time out if you take too long to get the headers to them, so maybe send the headers back faster, but like delay a couple of seconds, then send the headers, then trickle the rest at a few bytes per second.
Another way to do this is to use iptables. True story: `-m statistic --mode random --probability 0.5 -j DROP` does more or less what you would expect. This is what I did when Pawoo was flooding FSE with massive numbers of deletes, like as a kind of dopey rate-limiting ability. (Unintentional on their part: a few accounts with really long post history deleted themselves, and this causes Mastodon to send one delete per activity since the beginning of time to every server it has heard of...except the ones that it has blocked.)
> I figure that it the botnet notices when its banned and starts getting 403'd,
Basically zero of the scrapers that hit FSE do this. Boardreader.com didn't even notice when I started actively poisoning their data until about a week after I started including the phone number of the guy that was ignoring my emails in the data.
pistolero /
npub1ch…ysw60
2023-10-01 00:31:21
in reply to nevent1q…3ujn
Author Public Key
npub1ch8nj9yu4676fnwkzacu28mt4y002ezeryqyuhzfnzjw560sq5fqaysw60Published at
2023-10-01 00:31:21Event JSON
{
"id": "c962466af3053eeaa4834e828ddb25a91bb932db869b94e12fecf63587b02b92",
"pubkey": "c5cf39149caebda4cdd61771c51f6ba91ef5645919004e5c4998a4ea69f00512",
"created_at": 1696113081,
"kind": 1,
"tags": [
[
"p",
"192068fdc9629be317158fea8bbe16622bbf3f39b354caf84c31cacea79fcede",
"wss://relay.mostr.pub"
],
[
"e",
"a09540341f7d30dfd96bd353198879aa5ed9b6a2475fcd847d8e3e23d04e8f0d",
"wss://relay.mostr.pub",
"reply"
],
[
"proxy",
"https://freespeechextremist.com/objects/554e1643-b084-4a1d-a4ae-251def86364d",
"activitypub"
]
],
"content": "nostr:npub1rysx3lwfv2d7x9c43l4gh0skvg4m70eekd2v47zvx89vafulem0qav5m9t \n\n\u003e I'm getting my Nitter instanced scraped by a botnet that appears to be 100k IP large, they get fed in as fast as I ban them, but I don't believe they assign more than 100 IP to scraping at a time as to not DDOS the site,\n\nWEIRD\n\n\u003e one IP never doing a scrape under 7 seconds so rate limiting wont nab them.\n\nOut of curiosity, what UAs are they using? Tried SSL fingerprinting? You know why they'd be hitting your server, like did you check if DiscordBot or something is in your referrers, or someone linked to it from somewhere, or...?\n\n\u003e something I read about 10+ years ago, a sticky trap. I want to ensnare the bot into a perpetually open http request so that it never completes its loop,\n\nAh, okay, so you can do this pretty easily with nginx: you can forward to different backends conditionally with one of the (really badly documented) `if` directives. Set up a little script, listen/accept on one end, and then make a connection on the other to the actual upstream. So if your Nitter instance is running on localhost:4444, you have this script listen on localhost:4445. Have it relay all of the traffic upstream and then get the entire response (to avoid jamming up the real server), but trickle the response a few bytes at a time. Some clients time out if you take too long to get the headers to them, so maybe send the headers back faster, but like delay a couple of seconds, then send the headers, then trickle the rest at a few bytes per second.\n\nAnother way to do this is to use iptables. True story: `-m statistic --mode random --probability 0.5 -j DROP` does more or less what you would expect. This is what I did when Pawoo was flooding FSE with massive numbers of deletes, like as a kind of dopey rate-limiting ability. (Unintentional on their part: a few accounts with really long post history deleted themselves, and this causes Mastodon to send one delete per activity since the beginning of time to every server it has heard of...except the ones that it has blocked.)\n\n\u003e I figure that it the botnet notices when its banned and starts getting 403'd,\n\nBasically zero of the scrapers that hit FSE do this. Boardreader.com didn't even notice when I started actively poisoning their data until about a week after I started including the phone number of the guy that was ignoring my emails in the data.",
"sig": "1943088388a22a0b9afe8effa8dc956649db1ac8f464ceb929f9add611e956b9438af13c2d83ae8262eabc1ca93dacee99896a998d77054d18fdcdad77e82598"
}