Why cannot I forget he went with a man 😭 help?

My fiance I have been with for 6 years, had sexual relations with a man in the past. Which I didn't find out until I had had a baby with him. The worst part about this is that we used to go out in our teens. Then reconnected 15 years later in which HE TOLD ME that he split with his ex over confusions over his sexualtity. 

A few years later we ended up getting together and I never thought anything of it. Until his ex decided to tell me in order to split us up he had done stuff with a man. She hates me as I was his first love and he never got over that. My partner tells me it knocked him rotton after, had numerous showers and was discusted in himself. And closed that book for him and declares himself straight. We have the best sex life ever. I have now just had another baby... which he was told to terminate numerous times as had down syndrome. He's been born perfect. I know its me but every minute of every day I think about it. He is the most perfect romantic person I gave ever met. Works so hard for our family. Took my daughter in as his own. Even after me leaving him over 5 times to go back to my ex he waited every time. The perfect partner ANYBODY could wish for.. I want to put it behind me and enjoy the life we have got. Please kick me into shape people as im going to lose everything if I carry on like this due to my own head space. I know I deserve the comments I will get. I really do. I dont know if its hormones in I also have preeclamsia.