Is it wrong for me to not want to go back to this church ? ?

So I live next door to my ex boyfriends church, he no longer goes there, but I have so many memories there with him, and he put me through so much trauma that I don’t want to go back. The pastor and his wife, are always asking if I’m coming to one of their services again, I don’t have the hard to tell them that it hurts to be there. The last service I went to I ended up crying the whole time, thinking of my ex, and I don’t know what to do. I attend another church so it’s not like I’m not going to church, but I just don’t know if God wants me to keep going to my ex’s or not.