Jim
Jimstr

nostr relay sysadmin

Public Key

npub1an3nz7lczcunpdw6ltjst94hgzcxpppnk7zk3zr2nfcj4yd96kdse6twjd

NIP-05 Address

Profile Code

nprofile1qqswece300upvwfskhd04eg9j6m5pvrqssem0ptg3p4f5uf2jxjatxcpzfmhxue69uhk7enxvd5xz6tw9ec82cspzemhxue69uhky6t5vdhkjmn9wgh8xmmrd9skckw0pqs

Publishing to

offchain.pub

bitcoiner.social

Author Public Key

npub1an3nz7lczcunpdw6ltjst94hgzcxpppnk7zk3zr2nfcj4yd96kdse6twjd

Show more details

Last Notes

2026-03-06 19:05:07 GMT

- reply

nature is healing
https://blossom.primal.net/4e3e5237014f54f2dbf81d267b05435fa3ab4844b858284b06471e77964d25e0.png
#nevent1q…0v8h

2026-03-06 17:07:36 GMT

Somebody is holding 169,000 sockets open to the offchain.pub relay
Recently I started seeing Caddy spit out errors like this:
```json
{"level":"error","logger":"http.log.error","msg":"dial tcp 127.0.0.1:8080: connect: cannot assign requested address","status":502}
```
At first glance that looks weird. `127.0.0.1:8080` is localhost. It’s not a remote dependency. It’s Caddy trying to proxy to the relay process on the same machine and failing to even open the local TCP connection.
That usually means one thing: we’re running out of ephemeral source ports for the local proxy connection, or at least putting the box under enough socket pressure that new upstream dials start failing.
So I went looking.
## The first clue: way too many sockets
The first number that stood out was just total socket count:
```sh
netstat -antup | wc -l
169684
```
That is not a normal number for a relay with maybe hundreds of active human users at a time.
Then `ss -s` told the real story:
```text
Total: 169330
TCP: 169646 (estab 168773, closed 686, orphaned 167, timewait 679)
```
That changed the diagnosis pretty quickly.
This was not a classic SYN flood.
This was not a TIME_WAIT storm.
This was not some harmless burst of reconnects.
This was around **169 thousand established TCP sessions**.
If you’re proxying websocket traffic, every websocket is one TCP connection. So if you see hundreds of sockets from a single source, that’s hundreds of websocket connections, give or take some reconnect noise. A normal client should have one connection, maybe a few if they have multiple tabs, apps, or devices. Not hundreds.
## Parsing the sources correctly
My first pass at "top talkers" was wrong because IPv6 formatting makes lazy parsing lie to you. After fixing that, the top sources looked like this:
```sh
ss -Htn sport = :443 | awk '{print $5}' \
| sed -E 's/^\[([^]]+)\]:[0-9]+$/\1/; s/^([^:]+):[0-9]+$/\1/' \
| sort | uniq -c | sort -nr | head -40
```
And the output was full of things like:
```text
386 ::ffff:109.70.100.10
380 ::ffff:109.70.100.9
379 ::ffff:109.70.100.7
376 ::ffff:203.55.81.2
359 ::ffff:109.70.100.11
...
```
Those `::ffff:` addresses are IPv4-mapped clients hitting an IPv6 listener. A bunch of those ranges are Tor exits.
So, fine. That explains the shape of the traffic, but not the intent.
Maybe someone is trying to be annoying. Maybe it’s a badly behaved crawler. Maybe somebody pointed an agent at public relays and forgot to teach it what "enough" looks like. Maybe it’s an `openclaw` agent that lost its mind. I don’t especially care why. The machine still has to survive it.
## What the problem actually is
The important part is this:
- Caddy accepts inbound TLS/websocket connections on `:443`
- Caddy reverse proxies each one to `127.0.0.1:8080`
- each proxied websocket needs its own local TCP connection
- localhost connections still consume ephemeral source ports
- if enough pile up, new proxy dials start failing with `cannot assign requested address`
So the symptom was Caddy 502s, but the actual issue was **too many concurrent established sessions**.
## What I checked first
I did look at the usual TCP knobs.
The system already had:
```sh
sysctl net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_reuse = 2
sysctl net.ipv4.tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 60
```
And I already had a widened ephemeral port range:
```sh
sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 10000 65535
```
## What I changed
The fix I’m trying is simple and boring, which is usually how the good fixes look.
First, I kept the loopback-friendly TCP reuse behavior explicit and shortened FIN timeout a bit:
```nix
boot.kernel.sysctl = {
"net.ipv4.ip_local_port_range" = "10000 65535";
"net.ipv4.tcp_tw_reuse" = "2";
"net.ipv4.tcp_fin_timeout" = "15";
};
```
That is not the main defense, but it reduces some local socket pressure and makes the intent obvious in config.
The real mitigation is at the firewall.
I added a per-source concurrent connection cap for HTTPS with nftables via NixOS:
```nix
networking.firewall.extraInputRules = ''
tcp dport 443 ct state new meter https_conn_v4 { ip saddr ct count over 80 } drop
tcp dport 443 ct state new meter https_conn_v6 { ip6 saddr ct count over 80 } drop
'';
```
That means if one source already has more than 80 tracked connections, new ones get dropped.
Why `80`?
Because a normal human client should be nowhere near that number, but I’d rather start conservative and tighten later than accidentally clip legitimate weird-but-harmless behavior on day one.
I initially also added a per-source new-connection rate limit, then backed off that idea. Rate limiting is useful for reconnect storms and bursty abuse, but it wasn’t the direct failure mode here. The machine wasn’t drowning in SYNs. It was drowning in established sockets.
So the connection count cap is the thing that actually targets the problem I have. We'll see how it goes from there. If it's a legit attack, then they'll probably change the behavior and we'll have to adapt. But I'm still assuming it's a benign mistake like a misconfigured client at this point.

2026-01-08 04:03:15 GMT

We're back online!
RCA: Incognet's dedicated server panel license expired *while* we were installing a new OS on the relay after multiple years of running Debian. So I was unable to restore the server when I accidentally installed an EFI bootloader instead a BIOS bootloader. Woof. On top of that, I thought it was such an unlikely coincidence that I opened a case about it with only "Medium" priority, thinking if the NOC got to it within a day or two it'd be fine. After five days of no response (the promised support response time for Incognet is 1-5 days) I escalated to high priority and the issue got promptly resolved. So I take full responsibility for the downtime. Sorry folks!
I used to work in a data center NOC support, so I remember all too well how whiny support tickets can seem. That's why I try to avoid that tone and try to be generous with my patience when I (very rarely) actually need to bother NOC for technical support. In most cases I can self-service my way out of problems - this time was one of those rare exceptions.
More to come later about what's next for bitcoiner.social. Onward and upward. 🫡
#nevent1q…qdr3

2026-01-02 23:41:00 GMT

https://blossom.primal.net/882da85a03b7cdef8acb8d21452ca7c5c9c2a01dbc1ec008ea4651796654520c.png Yeah, I think reinstalling the OS on our blade server over a holiday weekend may not have been my best idea. The relay will be down until I get schedule on-site hands. The ticket is already a few hours old and I haven't heard back from the data center yet. 💀
Sorry for the down-time folks! In any case, I've got an offsite backup I made immediately prior to the reinstall attempt.
But hey, if things work out, I feel like I'll have a lot more time to dust off and fiddle with my nostr projects in 2026. Woo!

2025-07-10 15:54:12 GMT

in a couple weeks there'll be some free time and we're planning to rewrite the ansible playbooks for the relay as a nix config. I'll take a look at our WoT based filtering tiers and see if I can add any improvements. also going to see if I can set up an inbox relay endpoint. What features do you want from your relay?
the main thing I think we could improve is communicating the features that are already implemented. we published them before as long-form notes, but no one probably reads those. we should be publishing features on actual web pages. then no one can read those too! x_X

2024-10-22 22:18:21 GMT

- reply

Amen.
#nevent1q…yep0

2024-10-19 18:11:23 GMT

Brainstorming the next iteration on our web of trust model
https://bitcoiner.social/static/attachments/Ii83DA_weboftuss.png
* use postgres instead of sqlite
* admin UI for introducing new npubs into the web
* ollama assistant performs summarization and calls out any red flags
nostr feels like its in a goldilocks of freedom period. the right kind of plebs are already using it today. we gotta build our systems up before the 4th turning sends us all the zombies
pleb dev says he should have some time to work on this when he goes to Tabconf next week!

2024-10-05 15:16:19 GMT

- reply

Physical distance makes a big difference.

2024-09-05 18:50:16 GMT

- reply

And looks like we're already back. Only a few minutes of downtime after all

2024-09-05 15:10:05 GMT

Our dedicated server host is moving equipment to a new data center, so the relay will be down for a few hours today as their working. Here's their communication:
> What is happening?
>
> We will be physically migrating our hardware to a new rack within the same data center. The new rack is located two rows away from the current one, and the estimated time for this process is approximately two hours.
>
> Why is this necessary?
>
> This migration is part of our efforts to accommodate planned growth at this location. It will provide us with additional space and more control over our network and infrastructure, allowing us to bring this location to the same standards as our other POPs.
>
> How long will this take?
>
> While the physical relocation is expected to take two hours, we are setting a maintenance window of three hours to account for any potential delays. During this time, your dedicated server will be powered down, and you will experience a loss of connectivity. (This may not happen at exactly at 11:00 AM EST.) After the migration is complete, we will restore power and verify connectivity.

2024-07-25 17:46:00 GMT

- reply

should we set up a strfry filter to screen all replies from npubs outside our web of trust using llama3.1-3b?
wanted to do that for awhile but its been in the backlog

2024-07-20 16:43:53 GMT

- reply

👈

2024-07-16 15:30:57 GMT

Might switch the relay over to Ditto

2024-06-15 03:23:00 GMT

- reply

We don't keep logs but ip rate limits means we have your IP in a database for a few minutes after you post.

2024-06-09 21:28:16 GMT

Just initiated the DNS changes to migrate to our new server. We're keeping an eye on traffic starting to switch over now. Looking good so far.

2024-06-09 01:02:21 GMT

New server is staged so we'll tentatively schedule our relay migration tomorrow, assuming we don't drink too much tonight. Good evening!

2024-06-03 14:56:05 GMT

Waking up to see we have access to our new bare metal blade server for the relay. CPU: Intel Xeon E5-2690v2 ( 10 cores / 20 threads @ 3.00 GHz ) RAM: 64GB DDR3 ECC
I usually run Debian on servers, but maybe today's the day to put NixOS on a server. Hmm.
Good morning

2024-06-02 17:43:00 GMT

- reply

You're right, we can do better.
We've been keeping our smole relay fast and healthy is by periodically pruning events. Paid users like yourself still get longer duration. Occasionally when we see the usage spike, we dial up the pruning. Rinse repeat over the last year and now events expire within weeks rather than months like when we initially started. It's time for us to grow with nostr a bit, so we just paid to lease an affordable dedicated server in Kansas. This will increase the duration that we can keep posts, like more than 10x (last time we had a dedicated server we didn't prune at all). We backup our event store regularly, so when we migrate a lot of old events will get restored on this particular relay.
Our costs increase as well, so we're working on improving our web of trust model and eventually setting up subscriptions.
#nevent1q…4etp

2024-01-30 19:16:17 GMT

- reply

And we're back.

2024-01-30 07:25:20 GMT

- reply

Server is rebuilding the raid array currently, and should be back online afterwards. Couple hours maybe.

2024-01-30 01:28:52 GMT

A data center outage is in progress, the relay may be unavailable for a bit. We'll be back asap.

2024-01-21 04:58:19 GMT

Just completed some relay server maintenace today that changes our pruning strategy to keep events from friends-of-friends for a longer period.

2024-01-20 21:38:04 GMT

- reply

Fantastic! We have one too, no auth required currently.
ws://bitcoinr6de5lkvx4tpwdmzrdfdpla5sya2afwpcabjup2xpi5dulbad.onion

2024-01-05 22:29:26 GMT

@79f00d3f5a19ec806189fcab03c1be4ff81d18ee4f653c88fac41fe03570f432 137155 events, 268.3 MB bytes
that's a lotta zaps in our database 👏

2023-12-31 18:41:49 GMT

- reply

It's a temporary spam prevention policy.

2023-12-31 17:27:25 GMT

Happy New Year
We're just getting started here

2023-10-22 04:41:11 GMT

- reply

Because we'll never restrict the high utility event kinds (like nwc) and have long term pleb-sized plans for this relay.

2023-10-22 04:36:44 GMT

- reply

wss://bitcoiner.social if you want. We love you regardless.

2023-10-20 17:47:21 GMT

- reply

This list is a little dated

2023-10-13 20:40:31 GMT

Running @npub1048…x3nf
https://news.bitcoiner.social/

2023-10-11 17:44:18 GMT

We have opened the gates again. Public relay write access is available, with some restrictions intended to inhibit spam. All existing premium users of bitcoiner.social retain unrestricted write access.
Current policies are experimental and subject to change. Big thanks to @npub1dvd…0wh3 for testing some of them out in advance!
If you operate a broadcasting service, just provide us a stable source address to exclude from our rate limits.

2023-09-25 20:30:23 GMT

For those asking, our relay is temporarily private while we work on a new sustainability model. Aiming for a mixure of free or very inexpensive relay of notes (and other stuff). Most of the work up to now has been about making the existing relay faster and more stable. No point in charging for services if they suck, afterall! Our dev guy is working through a backlog trying to get to this.

2023-08-15 21:21:27 GMT

- reply

sup

2023-08-15 20:03:40 GMT

The mempool is a layer cake:
https://bitcoiner.social/static/attachments/Wi5noA_mempool-cake.png
That's the one-week chart. Monthly looks pretty similar. You know what this means!

2023-08-14 23:52:00 GMT

We just started running snort on bitcoiner.social
Now you can link directly to nostr events on this relay via our snort instance, such as your profile page:
https://snort.bitcoiner.social/p/npub1an3nz7lczcunpdw6ltjst94hgzcxpppnk7zk3zr2nfcj4yd96kdse6twjd
Big thanks to @npub1sn0…jtws
https://void.cat/d/89e8Dhgii9xnEBQfZnp7be.webp