Change baby sitters. He probably doesn't like her. And she sounds inadequate in responding so slowly.

Such behavior expresses emotions. He shouldn't be punished but try to find out what the emotion is and resolve the cause.