Oh, one way you can remedy that is by building an nprofile code for the person you want with the desired relays and adding that instead of just an npub.
You can do that easily with https://github.com/fiatjaf/nak -- nak encode nprofile (nak decode npub1... | jq -r .pubkey) -r relay.damus.io
Or with https://nak.nostr.com/ -- paste npub, click to add relay hint below (include the wss:// prefix).