I am not an expert, but I think that sometimes the only thing to do is walk away. You may not be aware of your son doing this to other people. He is four months old and I am sure that you are a terrific father. I think at that young age babies respond to their mothers, the mother is like the comfort zone. I know my daughter took a little while to "warm up" to her dad. Also is your son nursing? Nursing provides a special closeness between mother and child.

Don't lose confidence in yourself. I am sure that you are a great father and sometimes your wife may just need to calm him. Keep being involved and trying different things to see him through his tantrums, but if you need to hand him to your wife, don't take it too personally. Four months old is really little.

Amy

Well as amy said if your wife is nursing the baby will be more comfortable with her.
With my son he was the same way, I nursed him so he felt very close to me. I was given this adive and it work for my son's father,when it is nap time after he has been fead take your shirt off and his off and try to rock him to sleep.when you are skin to skin it make them feel very safe and secure, so atleast try that.

Shaina