I'd say losing the Freys, and ultimately your life, because you have some aversion to a woman who will be your wife in name alone for some chick you could find prettier at the local brothel isn't anything short of the most fucked up retarded decision.
Not to mention Frey's daughter ended up...