Should I leave?

My boyfriend does not get along with my family and I love having a strong relationship with them. This makes my life pretty difficult because both sides of my family and boyfriend feel controlling to overcome the other. I sometimes feel like my boyfriend wants to isolate me. He tries to only do sexual things im comfortable with and I try a lot but he gets upset if I don't do something he really wants me too. He also can get in horrible moods and bring me down. On the contrary he wants to make this work and tries so hard, always says sorry first spoils me in so many ways. There's so much I am devastated to lose. I am at a loss because when he makes it down to see me or vice versa (were long distance) we have so much fun. He's the sweetest person in the world when he's not in a mood and tries so hard to put aside some of his sexual interests to make me comfortable. I have this confusing paradox where I'm happiest with him and then the most upset with him. Any advice?