ex-boyfriend doesn't like for me to go to the gym?

My ex-boyfriend and I still talk all the time and we hang out. He goes out drinking all of the time and he even talks to another girl. We say we're just friends so I try my best not to hound him about what he's doing or what he's been doing. When we are around each other we don't know how to act like were not still together. But anyways, I've been going to the gym to work out and he gets so mad at me and starts acting like he's not going to talk to me any more. He's been trying to be better about going out and things like that so that maybe we can work things out and when I go and work out he says "I'm not even going to try anymore." Why does he care so much if I go? Is this normal? Why such double standards? I don't know what to do.