Well as amy said if your wife is nursing the baby will be more comfortable with her.
With my son he was the same way, I nursed him so he felt very close to me. I was given this adive and it work for my son's father,when it is nap time after he has been fead take your shirt off and his off and try to rock him to sleep.when you are skin to skin it make them feel very safe and secure, so atleast try that.

Shaina