Average middle-aged guy from various places in Ontario. Collector of various nerdly interests. Into #running, #books, #chess, #videogames. At least a little bit #buddhist. Trying to lead an examined life. I believe that we have the right to safe digital spaces where we can build communities free from corrosive capitalist influence. Working to build a sustainable, non-corporate web as a volunteer with #CoSocialCa Posts auto-delete after 2 months.
Public Key
npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3
Profile Code
nprofile1qqsx93dgyz6jzhpp5hwuwzzne6sqdfcp92nx3ew7my47zg0s9zgm2gqpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0dsjnhk77
Author Public Key
npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Show more details
Published at
2024-07-05T12:25:57+02:00 Event JSON
{
"id": "0c0bcc31e3c124f6a81995ea84078aceaca6af2de09d7cc87c3eba76c70b6b5b" ,
"pubkey": "62c5a820b5215c21a5ddc70853cea006a7012aa668e5ded92be121f02891b520" ,
"created_at": 1720175157 ,
"kind": 0 ,
"tags": [
[
"t",
"cosocialca"
],
[
"t",
"books"
],
[
"t",
"running"
],
[
"t",
"videogames"
],
[
"t",
"buddhist"
],
[
"t",
"chess"
],
[
"proxy",
"https://cosocial.ca/users/mick",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://cosocial.ca/users/mick",
"pink.momostr"
],
[
"expiration",
"1722767157"
]
],
"content": "{\"name\":\"Mick 🇨🇦\",\"about\":\"Average middle-aged guy from various places in Ontario. \\n\\nCollector of various nerdly interests.\\n\\nInto #running, #books, #chess, #videogames. \\n\\nAt least a little bit #buddhist. Trying to lead an examined life. \\n\\nI believe that we have the right to safe digital spaces where we can build communities free from corrosive capitalist influence.\\n\\nWorking to build a sustainable, non-corporate web as a volunteer with #CoSocialCa\\n\\nPosts auto-delete after 2 months.\",\"website\":\"https://cosocial.ca/@mick\",\"picture\":\"https://media.cosocial.ca/accounts/avatars/110/058/650/635/486/844/original/687f80524df51ae5.jpeg\",\"banner\":\"https://media.cosocial.ca/accounts/headers/110/058/650/635/486/844/original/54b6e9538d5e3308.jpeg\",\"nip05\":\"[email protected] \"}" ,
"sig": "4b1b1f4a4964ae445c86950014ad2e13f7089497baf6485ba064a4f91899760166108acdd169e05a21d2f4a0bba66d98c66b195c5827d6f45c6ab7ab24296456"
}
Last Notes npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 So much excellent writing in this piece by David Roth. Another banger: “A politics whose most fundamental idea is Make Progress Stop Happening would inevitably find itself fetishizing the torment of having to live in a world in which other people, who are not even you, are somehow supposed to matter just as much.” #uspol https://cosocial.ca/@mick/112949352874573609 npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 I look forward to reading a blog post about the process when you get there. npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 nice shot! npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 nice shadow! npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 Bacchus would *definitely* have been dtf tho Lewd tourist antics on Florence Bacchus statue lead to outrage https://www.bbc.com/news/articles/cw5yky19623o npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 Prometheus, and this sidekiq-exporter https://github.com/Strech/sidekiq-prometheus-exporter npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 This strikes me as an issue. We have the capacity to run 40 workers (following the change I made last week, documented earlier in this thread.) We have fairly huge backlog of pull queue jobs. Why aren’t we running every available worker to clear this backlog? 🤔 It might be necessary to designate some threads specifically for the pull queue in order to keep up with whatever is going on here, but I am open to suggestions. #MastoAdmin https://media.cosocial.ca/media_attachments/files/112/481/766/435/082/419/original/12a8f15d76f03371.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 Well, we’re not out of the woods yet. We fell behind by less than a minute for most of the day yesterday, with some brief periods where we were slower still. The droplet is showing no signs of stress with the increased Sidekiq threads, so I can toss a bit more hardware at the problem and see if we can reach equilibrium again. Better would be to get a clearer picture of what’s going on here. Maybe we need to do both of these things! 10/? https://media.cosocial.ca/media_attachments/files/112/479/571/178/559/668/original/cd97e96dc41ec800.png https://media.cosocial.ca/media_attachments/files/112/479/571/484/364/234/original/9b01c0fe0d2526eb.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 This looks better! Pull queue never got more than 41 seconds behind and that was only briefly. I still am not clear on what has contributed to these spikes, so there’s no way of knowing for sure that the changes made yesterday are sufficient to keep our queues clear and up-to-date, but this looks promising. 9/? https://media.cosocial.ca/media_attachments/files/112/465/475/876/577/639/original/8616d38782d4db20.png https://media.cosocial.ca/media_attachments/files/112/465/476/068/770/996/original/70754e0acb6044d0.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 Aaaand we’re good. 🎉 I’ll keep an eye on things over the next days and week and see if this has any measurable impact on performance one way or the other. And that’s enough recreational server maintenance for one Friday night. 🤓 8/? https://media.cosocial.ca/media_attachments/files/112/460/094/059/327/394/original/96764e528fb2d3d5.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 No signs of trouble. Everything still hunky-dory in staging. On to production. If this is the last post you read from our server then something has gone very wrong. 😅 7/? npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 Staging server only has 2 GB of RAM but it also has virtually no queue activity so let’s give it a shot. Having confirmed that we have sufficient resources to accommodate the increase and then picked a number out of hat, I’m going to increase the number of threads to 40. 6/? https://media.cosocial.ca/media_attachments/files/112/460/007/536/250/495/original/4b7db19bbab92afc.png https://media.cosocial.ca/media_attachments/files/112/460/008/023/393/363/original/e7bc8ee1055c4a0a.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 That article from DigitalOcean suggests that 10-15 threads = 1 GB of RAM. We also need to give each thread its own DB connection. In staging the DB is local, so we don’t need to worry too much about a few extra connections. In production, we’re connected to a DB pool that will funnel the extra connections into a smaller number of connections to the DB. Our Database server still has oodles of capacity to keep up with all of this. 5/? https://media.cosocial.ca/media_attachments/files/112/459/965/366/664/268/original/c7942847ad5673a6.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 We’ll do this staging first, because I am a responsible sysadmin (and I am only ever half sure I know what I’m doing.) We’re running the default config that came with our DigitalOcean droplet, which as a single sidekiq service running 25 threads. 4/? https://media.cosocial.ca/media_attachments/files/112/459/886/549/602/965/original/1ca67e12ef6edb9d.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 For anyone interested in understanding the guts of Mastodon, I have found this article from Digital Ocean very helpful: https://www.digitalocean.com/community/tutorials/how-to-scale-your-mastodon-server#perfecting-sidekiq-queues Eventually we’ll grow so big that we’ll need oodles of sidekiq queues and we’ll want to be able to customize how many types of them we want, and will run them as jobs across multiple servers and so-on. But for now I’m just going to make the number of threads slightly bigger and see what happens. 3/? npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 That would be helpful. I’d love to be better able to interpret weird traffic spikes like this. Without being creepy about it. 😅 npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 This is great! It’s exciting to see our community growing. I’m going to make a simple change to see if we can better keep up. The system that we’re running on has plenty of headroom for more sidekiq threads. 2/? https://media.cosocial.ca/media_attachments/files/112/459/824/661/617/895/original/df65e4de62ac678c.png npub1vtz6sg94y9wzrfwacuy98n4qq6nsz24xdrjaakftuyslq2y3k5sqvrzqq3 Mick 🇨🇦 For the first time the #CoSocialCa Mastodon server has started to struggle just a little bit to keep up with the flow of the Fediverse. We’ve usually been “push” heavy but we’ve started to see some spikes in “pull” queue latency. The worst of these spikes was today, where we fell behind by at least a couple minutes for most of the afternoon. 1/? #MastoAdmin #CoSocialTechOps https://media.cosocial.ca/media_attachments/files/112/459/768/298/723/783/original/df163724123aded4.png https://media.cosocial.ca/media_attachments/files/112/459/768/565/485/567/original/6dde9cac694878bd.png