how can i get my really annoying friend to leave and never come back subtly without the whole house knowing?

he comes round my house at least every Saturday to play D'n'D of which he is the host (DM) he kicked me from the group on multiple occasions simply because me and my ex broke up, and only allowed me to join when we were together (temporarily) again, i have tried to make friends with him by helping his D'n'D but he's kicked me out again and now calls me autistic and makes remarks on my mathematical skills in my own home, as we get into rows quite often it is obvious we can't "just get along" and because he is close friends with the most of my house hold, i can't just tell him to leave, he even got my brother to "his" point of view, he has to go, i can't spend every Saturday in my room all day or go out as i have nowhere to go, and my other friends would get quite peeved after awhile :/ i need either a fiendishly clever plan to get him to leave and not come back without the household disowning me or a way to be friends with him again