No, I siege a town, wait for the big party to come in, and defeat them, releasing everyone after.
Repeat this 5-10 times and your relations will be 40+, at which point you can start recruiting.
I only siege to bait them, not with the intent of taking the town.