Hmm. I don't think he felt bad about dodging parole, the first song clearly indicates that he thinks that slaving for twenty years for stealing a loaf of bread to feed your family is unfair. He only felt bad when the other guy got accused in his place and just generally uncomfy that he needs to hide his past.
Also don't you just love the well written slow burn realistic romance there?