Do you get much one on one time with him? It can't be easy with so many other commitment. Maybe a male role model in addition to his father would help. Sometimes relating to a figure outside his immediate circle may help. Does he have any younger Uncles or do you know any responsible young males he can do an activity with like - fishing or tennis or such like?
As for the you are too soft line - I don't think one can be too soft with kids - I do think that you cannot give into bad behaviour and it is best to ignore negative comments and praise positive comments. Hang in there.