Physical authentication tokens are only as good as the recovery mechanism when people lose one. Make the recovery too easy and attackers can bypass your spiffy token. Make the recovery too difficult and you can lock yourself out.
The sole context where I’ve seen physical authentication tokens work well is at companies with well staffed and highly responsive IT teams.
So before wiring anything up to that YubiKey, ask yourself what happens when you lose it.