He wont give me my separation!!?

My husband of 3 yrs, formerly abusive slightly physical and all emotional and verbal wont let me take my 1 yr old when i go live in my own apartment for a whole week!! Says she shouldn't be bounced around and he says we need to do it weekly....what he says goes, no negotiations. He is being a jerk?? Or is he right about this?? Plus i cant seem to get out of here cause he is saying that things arent that bad anymore, but iam miserable and i told him this. we have tried counseling but i cant get past what he did..we had a separation a yr ago after he abused me...i went back after counseling and he hasnt abused me since...but i told him iam miserable but he doesnt seem to thing that is enough! He says if i leave again than he is filing for divorce and the baby is staying with him till next weekend. She needs me! I tried offering two days on and off but he wouldnt hear of it!