Short visit with the "MOTHER" there, but absolutely not the whole weekend without the mother. And as long as she is breast feeding, no overnights with Daddy. There will be plenty of time for bonding during the day and after weaning.

Edited to add that there needs to be a court ordered custody arrangement immediately.  Funny things happen when there is no custody orders. and even then it can get hectic.