I got some feedback from Adrian Vovk (npub1qeh…ye8a)
You can fix the issue by disabling automatic resizing and setting a fixed sized for your home directory.
The fix for auto resize being slow would be https://gitlab.gnome.org/Teams/STF/homed/-/issues/7