We borrowed against a 401k. There's no penalty if you repay it on the plan, which is typically 5 years. I think for primary residences it can be 10.
Roth money, you can withdraw your contributions at any time. There's only a penalty if you withdraw gains. Of course, the opportunity cost is...