This was basically a love triangle between the three of you; I would guess that your EX quite enjoyed "keeping" you and still being able to start something new. Triangles are never good, there is always someone who gets the short end, and it sounds like it may have been you and your daughter? I’m only going by the fact that you said you let him in and out of you life (and your daughters) for so many years.

He is the adult and if he wants a relationship with his child he would have to initiate it, I would talk to your daughter and just explain the situation as best you can, especially if he is like you say "devoted to her children" even though they are not his. THis must make your daughter feel horrible.

I commend you for leaving this situation; hopefully you can find a love to devote himself to you and your daughter. "That time has come"