That's what I grew up calling gravel. I remember walking on it barefoot! Ouchy, ouchy!
I've heard that lavender does quite well in places like Montana that get extremely cold ... but are also very dry. There is a woman here in Maine with a huge lavender farm; that would be a fun visit because she invites people to just come and sit and enjoy.