nirik :fedora: :redhat:

Sysadmin shouting at clouds. #fedora #redhat

Public Key

npub1hm6w3hk9x5jj25ljq68ldluukgqr29fq8p83u2ut7ysvyd9se0vsuqedjd

NIP-05 Address

Profile Code

nprofile1qqstaa8gmmzn2ff920eqdrlkl7wtyqp4z5srsnc79w9lzgxzxjcvhkgpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0ds3625tm

Publishing to

relay.damus.io

nos.lol

Author Public Key

npub1hm6w3hk9x5jj25ljq68ldluukgqr29fq8p83u2ut7ysvyd9se0vsuqedjd

Show more details

Last Notes

2024-06-06 19:33:54

This morning:
Critical, severity one (home) infrastructure incident:
My coffee machine is leaking.
Sev1! all hands on deck!
(took it apart some and cleaned everything and the leak seems much smaller now, but something is still leaking)

2024-04-30 00:23:31

gpg is the worst.
Generating a new set of keys on the new nitrokey3... saved my pin/admin pin in pass. Generates for a long while, then needs the pin to make the signing key... but I can't get the pin from pass while there is a gpg prompt up for the nitrokey. :( While trynig to figure out what arcane gpg command line will not use the same pinentry, it times out and I have to start again.

2024-04-19 04:20:36

- reply

* Got another server finally all green finally after a tech showed up and replaced the motherboard. Hurray. It started having problems in january. :( It's been a long road, but finally over.
* Pushed out a updated koji package to builders because the upstream patch to fix something isn't working as I expect. Going back to an older version that I know works. Should fix unsigned packages in f40+ containers.

2024-04-19 04:18:49

Today:
* We are go for f40 release next tuesday! Hurray. I'm a bit surprised no blockers came along, but happy too.
* Worked with datacenter onsite folks to get another machine setup, a CMOS battery replaced in another one, and confirm another one is set except for switch ports.
* Two servers with memory errors, but will reboot after f40 is out.
* One server has some fan alerts, but it's going away soon, so just going to ignore it and move things off it.
<next toot>

2024-04-18 03:53:47

- reply

@npub12h3…9qyg everyone says that, but it's really kind of hard sometimes. Some of them are actually asking for information or expressing some interesting viewpoint. Oh well.

2024-04-18 03:48:06

Today in fedora infraland:
* Moved some staging vm's so we could remove 2 older virthosts and replace with one new one. 'virsh migrate' is so cool. ;)
* fedora_koji volume hit max space and started tossing NOSPACE errors. Deleted 6TB snapshot. Soon will archive off things.
* Re-installed a new kernel test machine, still need switch ports set to hand it off to kernel folks.
* Looked at another new machine, but it's on the wrong vlans, onsite people will work on it tomorrow.
<cont in next toot>

2024-04-17 03:08:06

Monday:
* More hardware fun with server showing bad memory stick (it's the motherboard).
* Confirmed all 3 new severs we want to setup this week are actually there ready to be racked.
* F40 blocker review meeting
* Long FESCo meeting
* Setup one of the 3 above servers, installed, ansiblized it.
* usual tickets/meetings/posts/etc

2024-04-14 00:34:02

Picked up a nitrokey 3c to play around with. Started messing with it and got to gpg stuff. gpg's interface is... not something I like at all. Dunno if I will power though or not.
The main reason I need it right now is that I want a ssh key where the private key is not on my laptop ever, and can't use ecdsa-sk on all machines yet.

2024-04-12 03:43:47

Today:
* Meetings, include f40 go/nogo meeting (it was no go due to some hardware specific blockers).
* Fixed up some failed updates pushes and pushed some more stable f40 updates from QE request
* Gathered info and wrote up a doc on datacenter work next week. We have 3 machines to rack in and 5 to remove. Hurray higher density!
* Stupid koji hub locked up again. ;(
* Installed and setup a new virthost
* Cleaned space to fix koji space alerts. :( New storage soon so I can archive things.

2024-04-12 02:27:40

Wed (yesterday I think it was, but so long ago):
* Synced F40-RC-1.12, found an error in image naming. ;(
* Bunch of meetings. Like 5 of em. Some good / productive ones though!
* Caught up a lot on emails and tickets. Had 35 marked to reply to, down to 18!
* Synced new rc-1.13. This one looks much better!

2024-04-11 23:25:29

Behind again, but instead of dumping a bunch of things at once, lets spread them out some. ;)
So, Monday:
* Updated builders to pick up 2 kiwi files that were in the infra rpm, but hadn't been applied yet. (Mostly because I thought the changes we hub only)
* Poked around with snapshot app on my laptop to help triage bugs with it. It actually works fine booted from a f40 nightly, so my issues are rawhide only.
* Created some aws vpcs to help qa testing stuff.
* Usual meetings/tickets/etc.

2024-04-08 03:30:44

I'm not a super IPA fan, but a cold one sure is nice after walking around the property and fighting with the blackberries. :)
(Beak Breaker double IPA)

2024-04-06 21:09:41

Got behind again. ;)
Thursday:
* Sorted out a weird openshift problem that turned out to be someone (me) not having added one of the compute nodes to export-policy. ;(
* Adjusted some f40 flatpak tags to get f40 flatpaks building.
* Rebooted some builders that were not responding. These old x86_64 builders may finally just be ready for retirement.
* Usual small stuff all over the map
<exciting friday in next toot!>

2024-04-04 20:57:18

- reply

@npub1vz9…jh9n yep. I know...
I'm one of those weirdos that switches desktops every reboot.
I can get my work done just fine in any of them...

2024-04-04 19:41:31

Hey look... 13 years ago today I started working at #redhat
Here's to the next 13. :)

2024-04-04 03:03:43

Today:
* Looked at proxy where httpd seemed to just get stuck. Still looking.
* Helped debug hardware issue with a server. Looks like bad memory slot/MB
* Cleaned up snapshots on koji volume again. Need to move more off it when our new storage is ready (very soon I hope!)
* Merged a ton of pr's
* Created some NFS volumes
* Processed a few tickets, slowly working thru them.
* Completely reinstalled my laptop, just for paranoia sake.
* so much email...

2024-04-03 03:50:19

- reply

* Went to the dentist. Deep cleaning on the left side. :( Right side next week. No fun, but needs doing.
* Caught up on emails, etc

2024-04-03 03:48:34

Monday:
* More xz fallout/coordination
* cleaned up some ansible config
* Tracked down why package retirements were broken since Mar4th. Was a hook we disabled. Re-enabled it for now until we can fix the consumer. ;( Still need to clean up the missed ones
* Reinstalled one of the koji hubs
* reinstalled kojipkgs01/02
* Applied updates to some isolated servers.
* Some tickets, meetings, etc
Tuesday:
* Pushed some last min ansible changes
* Started f40 final infra freeze
<next toot>

2024-03-31 04:01:07

- reply

@npub1c9r…99yp don't think so. ;) really the odometer is just under 5k miles.

2024-03-31 03:58:04

Wow. Thanks Toyota app... I never thought I would get this many miles...
https://cdn.fosstodon.org/media_attachments/files/112/187/817/544/251/974/original/93e1d6180f7f6d06.png

2024-03-30 03:34:27

- reply

* Applied updates to a bunch of machines.
* Upgraded ipa cluster hosts when those updates updated ipa.
* koji02 hub became unresponsive, rebooted it. Luckily it didn't break the rawhide/f40 composes I was trying to push with the xz downgrade.
It's been a long 2 days... :(
Hopefully it will be a quiet weekend....

2024-03-30 03:31:49

Thursday:
* Helped fix review-stats tokens.
* finally retired our OSBS clusters. Been a long time coming. They were running el7 and fedora (cough). x86_64 and aarch64. 14 vm's shutdown.
* Wrote up a thread on doing a hackfest at flock2024: https://discussion.fedoraproject.org/t/planning-for-infra-and-releng-hackfest-at-flock-2024/110244
Please chime in there!
* reinstalled our openshift control nodes (were rhel7, now rhel9)
* Started helping get a update pushed out.
Today:
* More work on pushing... xz updates.
* Fixed backups on a ipa cluster host.
<next>

2024-03-29 20:39:20

- reply

@npub12zn…z0zw Fedora 40 had the affected version, but it wasn't actually affected/active. (But you should apply the downgrade upgrade anyhow, just to be safe).

2024-03-28 02:26:44

Got behind again. :(
Monday:
* Gor Fedora 40 Beta torrents all working. We have some horrible old scripts that need re-writing.
* Fixed some f40 updates that hit a corner case in bodhi.
* Fixed the fedora.im cert that expired. ;( Will fix that from happening again too.
* Did some prep for beta release
Tuesday:
* F40 beta is out! Should be pretty darn solid. I am very glad we waited a week to land more fixes.
* Cleaned up some old ansible facts.
* Some AWS and other tickets.
<next toot>

2024-03-24 23:53:14

- reply

@npub1c9r…99yp All of them! https://github.com/0xD4ni/cats

2024-03-24 18:13:07

- reply

@npub1pzn…y2as I like mine pretty well... ;) Happy to answer any questions if you hit anything on it!

2024-02-07 20:00:10

Is there a word for when you have a sudden panic about something that could be very bad, then you check and see that past you already did the right thing and it's completely fine? :)

2024-01-22 00:55:00

cloudflare seems to be putting me in a infinite 'verify you are a human' thing with firefox (works fine with chrome*).
Anyone else seeing this?

2024-01-16 02:31:19

- reply

@npub1c9r…99yp I wonder if this was before or after geddy was kicked out and readded. ;)

2024-01-15 22:40:37

Looks like our luck ran out and we got hit with a power outage. ;)
Had to chip ice out of my generator connector, but got the generator up and running fine.
Unfortuately, the power it provides isn't stable enough for my UPSes, so my main server/gateway/wireless is down. I am on via my phone hotspot.
So, if you are pinging me in matrix or sending me emails, you will likely need to wait until the power comes back. :)
Time to go look at inverter generators again.

2024-01-11 02:13:54

today in infra:
* Added 500GB to our lookaside cache volume.
* Moved proxy04 off ibiblio01 to 02, and then shut 01 down and asked it to be decomissioned (it's out of warentee/old). (one less rhel7 machine!)
* looked at a flatpak-cache app and provided some feedback.
* Wrote up current status of all our rhel7 stuff: https://discussion.fedoraproject.org/t/rhel7-status-in-fedora-infra-early-2024-edition/101411
* Usual meetings, tickets, pr's, etc.
* Rebooted my laptop with 6.8.0-0.rc0.20240110gitab27740f7665.4.fc40 and...it works fine.

2024-01-09 01:29:50

aaargh. my coffee machine is leaking and I can't see why. :( I guess it may be time to get a new one... had this one for like 10 years. So many coffes.

2023-12-29 22:02:59

Just spent ~30min trying to figure out why sound wouldn't work at all on my new kodi box. Turns out my stupid tv decided to just... ignore HDMI sound. Rebooted it and... everything works as expected now.

2023-12-27 23:33:42

- reply

@npub1c9r…99yp Thats just the start... but emails, source code, logs, media, copy of my main server, etc etc.
The media I would share, but the rest is not something I would want to share.

2023-12-27 23:31:18

Copied almost 3TB of stuff to the new backup drive I got for xmas:
/dev/mapper/external-backup 13T 2.9T 10T 23% /srv/external-backup
With this I will have backups, and a backup of all the backups. Now I just need the remote backup of the backups (coming soon).
(Can you tell I'm a paranoid sysadmin? :)

2023-12-15 06:21:14

This is the best ethernet interface name yet:
enp193s0f3u1u4

2023-12-09 03:20:34

- reply

* Those problems made it so bodhi never defined/ran it's 'mid' pod to do schema upgrades (which worked because we had a playbook doing it manually). I fixed all that and it should now do that all correctly, and has it's new timeout.
* Am now on PTO until january... but of course since I like working on community things, I will still be around from time to time and in case of any outages, etc.
Happy holidays everyone!

2023-12-09 03:18:21

- reply

* Went to file a koji bug on it not killing too long running builds, to learn we already did file a bug on that 2 years ago. ;(
* Meeting on no meeting friday... but... it's my very last scheduled meeting for 2023! :)
* Tried to adjust db01 and db-koji01 to handle load spikes better with mixed luck. So many postgresql tuning knobs. ;(
* Tried to make a 1 character change in bodhi config (from 10s to 15s for a health check) and got sucked into fixing a bunch of indentation problems...
2/

2023-12-09 03:15:36

Yesterday and today:
* Got koschei back up and running. Deleted f37 collection there since it's EOL and tweaked things to keep it up and non crashing.
* Reissued the new *.fedoraproject.org cert because somehow it was against a 2048bit root cert and some people seem to want to set FUTURE crypto and still want to get metalinks for updates. It should now work again, for whatever values of work. You are in for a pretty broken internet with FUTURE right now.
1/

2023-12-03 00:32:11

At least on the plus side since things were down, I put my new video card into the server. Didnt work, but when I swapped it with a network card it did. Of course swapping the network card made 'predictable network names' change and fail to bring up my internal network.

2023-11-25 00:13:50

I see #fractal 5.0 is out. Congrats to all who worked on it! (The gtk4 matrix client).
Sadly, I'm in the same boat as @npub12h3…9qyg here, the density is just really not good for heavy users. Hopefully they will decide to address https://gitlab.gnome.org/GNOME/fractal/-/issues/1072 at some point. :(
Still using #nheko here. (Although I really wish they would do a release, there's a ton of fixes/improvements in git).

2023-11-19 22:38:40

- reply

* Spent much of last saturday afternoon trying to see why rawhide compose failed. Turns out the condor package started bundling libfmt and providing it and... that broke the world. :( Untagged, filed bug.
* Updated and rebooted all our machines. Some f38->f39 moves, some rhel8->rhel9 moves. About 450 instances overall. Took most of tuesday/wed.
* Worked to get a core out of weird aarch64 livemedia failures. (SIGILL in python). rhbz#2247319
/3

2023-11-19 22:34:11

- reply

* Cleaned up a bunch of koji build targets. There were a bunch of leftover early modularity/mbs ones. ;(
* Upgraded some matrix rooms. Both to get new room features, but also to drop old IRC bridge puppets and fix some federation issues (hopefully).
* F39 release party and 20th annv. I had a short 15m block at the end to talk and (while rambling) mentioned some folks who really helped me: @npub1usy…5y4m and skvidal. :(
* Got resultsdb setup to allow coreos testing results.
/2

2023-11-19 22:31:11

So, I'm 2 weeks behind on my "what I have been up to". ;) So, here goes a catchup thread. :)
Starting Nov 6th
* Finally fixed the robots.txt on koji to deny crawling. db load has been much much happier since.
* Got the last 3 missing artifacts respun for f39 release.
* F39 release! Hurray! so glad to finally release it.
* Unfroze after release. Merged a zillion pent up PR's and changes.
* Updated openshift clusters to latest.
/1

2023-10-31 02:09:33

- reply

@npub1hlt…ck4e https://frame.work/products/bezel?v=FRANCBCP07
I don't recall seeing it in the past... not sure when it was added. :)

2023-10-31 01:58:11

- reply

@npub1hlt…ck4e Not the same, but you can get a red bezel for framework laptops. ;)

2023-10-25 20:58:42

WOAH! I needed to print something (for the first time in years). So, I turned on the printer, and I submitted a print from my laptop and... it printed! It's readable even! I'm completely shocked.

2023-10-16 19:22:54

- reply

@npub1w94…0wl6 Only 3? :)

2023-10-07 18:46:08

Sheesh, behind again on my recent fedora infra adventures. :(
In the last while:
* Made some good progress on new sigul in staging, almost set there.
* Fell into a rabbit hole trying to get a matrix room back to normal. Still not 100%, odd federation issues, but at least more moderators.
* Finally after a number of rounds, got fedora 39 flatpak's all able to build and be pushed out. Hurray. This setup is soooo much better than before.
<next>

2023-10-04 20:05:53

What is the logic behind people who call you, get voice mail, then don't leave a message and... call you again (repeat several times in a row some cases).
Do they think you just didn't get to the phone in time? That you will really want to answer it if they call again? That they think you will guess it's important and answer if they call back a few times in a row?

2023-09-15 21:08:54

Hey look, now that F39 beta is signed off on, time to push all those stable updates that were waiting on the freeze:
Push these 717 updates? [y/N]: y
#fedora

2023-08-23 02:38:15

I've finally gotten caught up enough to actually look at things on my todo list instead of just adding them without looking. ;) Hurray?

2023-08-20 18:19:56

- reply

@npub1sjd…n8pk alas, it doesnt work here. it pops up a new dialong telling me I have to login. ;(

2023-08-19 23:53:01

- reply

@npub1usy…5y4m yeah. I just wanted to know when the small local bakery was open unitl. oh well...

2023-08-19 23:35:54

- reply

@npub1uje…yxv9 It used to be you could just click no thanks, but sometime in the last few years it changed to require it. :(

2023-08-19 23:32:39

Dear small businesses: Please don't use facebook as your primary web page. It now requires you to login and some of us have no interest in doing so.

2023-08-09 18:59:19

- reply

@npub18yv…xxyn I've actually found interesting how much more active the matrix side is than the IRC side. I suppose it all depends on channels/rooms/communities.