i have a 10yr old son, he to can be very loving but i have noticed his aggression towards his siblings(girl8 and son5), i have however put it down to hormones, i think sometimes we find it had to remember ourselves going through that period in lfe. i get told i'm ruining his life etc..etc.. dont beat yourself up over it its not worth it, you sound a pretty grounded person and kids are a challenge none of us are really prepared for, hopefully his behaviour is something he'll grow out of, best of luck my friend and remember your not alone

Do you get much one on one time with him? It can't be easy with so many other commitment. Maybe a male role model in addition to his father would help. Sometimes relating to a figure outside his immediate circle may help. Does he have any younger Uncles or do you know any responsible young males he can do an activity with like - fishing or tennis or such like?
As for the you are too soft line - I don't think one can be too soft with kids - I do think that you cannot give into bad behaviour and it is best to ignore negative comments and praise positive comments. Hang in there.