They don't know that that's their "main force" though, they think the whole wall is manned, that's why they went out and killed the deserting crows so they wouldn't spill the beans on their lack of numbers.
And they were just testing defenses, you never throw your whole force out there right...