I'm confused that you and your ex still do things together when you are separated! My parents divorced when I was 9. Thankfully they kept things fairly amicable.
I think your're confusing your son by still acting like one family unit which is giving him false hope that you may get back together. In my opinion you need to have a routine and that has to involve your ex taking both children or not at all. It is totally unjust to leave your 5yr behind, she will feel rejected and the repercussions will materialise in later life.
If your're intending to divorce you need to make a clear break for your own sake and the children. Things will become awkward should either of you meet new partners and I don't think they'd be a fan of these joint activities.
A routine should be made. My dad saw me and my younger brother every fortnight and eventually his new partner (now my stepmum) and her kids joined in the fortnightly activities. I recall being shocked and upset when my parents separated as I never saw it coming and nor had any of my friends experienced divorce. However due to a routine I received no mixed messages and fairly quickly adjusted to the new routine. My parents have never asked me to take sides which is a credit to them in retrospect.
I'm not saying that you shouldn't spend odd occasions together e.g. birthdays, xmas etc. However I don't think if you had a new partner that he would appreciate your ex coming round for dinner. Your son would resent your new partner if he believed he was the reason for your son not being allowed to spend time with his dad at your house anymore.
Your son is capable of being well adjusted if he sees his father on a routine basis. Quality time with dad is better than quantity.
You need to be brave and have a reasoned discussion with your ex and explain that he is welcome in your home to pick up the kids but there'll be no more dinners and joint activities. You can't carry on the way you are as your're confusing the kids and it will make life easier should either of you meet new partners. Hope what I'm saying makes sense.

thank you for your response...

your words make a lot of sense to me and I have heard them all along. The problem is that my ex will simply not obey by these rules...He says his house is too small to take both of them...which is not true...that is why i sometimes think fleeing is better...so that we don't have these confused messages.

But you are right...I need to be braver....