In feudal times, most of the people were bound to the feudal's property and land. To move, you had to get an approval of both your's and target location feudal. Usually, paying tributes to both of them - to let you go and let you in.
Truth is that in much less densly populated europe, you could find a place to live away from civilization. The price was that you could not leverage full benefits of labour division and everything yourself.