That would have required political savvy and courage, but you're right.
Personally I feel the rules were intended to benefit Biden, but it didn't pan out that way. Trump is good at debates, unless you have a candidate who can go up against him, they're better off staying home.
Admittedly Biden was better against Trump last time around, but Trumps a bully, and that's hard to beat.