I remember this show from way back when. Had a book that showed how things worked down there. The doozer's are people who have to constantly stay active and do so through construction. Their building materials are based off of vegetables, most of which are radishes which are the fraggles favorite food. They get it from the garden of some guy close to the surface who has to always fight off the fraggles who steal them.
Here's the kicker from the story though. The Doozer's actually turn into the fraggles when they stop working. They grow larger and more gluttonous and eventually mutate in a word.