How to get a guy to stop holding grudges about the past?

Well I have been with my bf for almost 4 years and a year ago we got into a huge fight and we decided to break up...it was for about 2 months..then got together again but broke up again for another month. We were each seeing other people and he got mad at me because I had sex with someone but he didn't. And he says "if you truly loved me you wouldn't have had sex" And that's not true at all I was just trying to move on is all....just as I'm sure he was... It's not my fault he couldn't have sex because he "loved me too much" is what he says..

We don't live together right now but we see each other every weekend and we have the best weekends ever, but then during the week when we're not together we talk and have decent convos as we always have but he seems to pick fights with me... and it's always about bringing up guys I was with but when I was with guys it was when we were BROKEN UP! And he thinks I was with more guys than I told him. Why is he doing this? Is it out of boredom or what? He always tells me "im over it babe, im over it" but then 2 days later it starts again... ughhghg its so annoying...any advice on what this is all about?

We don't wanna break up as we have realized that we truly want a life together and we really are a good couple I just can't get him to get over this past **** that isnt even irrelevant!