I'd keep them 4-6 weeks to get over the castration stress and to get some gains on them to offset the shrink you got bringing them home plus the shrink you'll get bringing them back to the barn again.  While you're at it, you can follow a vaccination plan and truly precondition them.