How should I deal with this freaking work drama?

I have a co-worker that I am definitely clashing with! Just a little background, I have only had the job for 4 months, I moved a long way to take it, I got a graduate degree to have it, and there aren't a lot of jobs around like it so I can't just "go find another job". So this guy I work with honestly has nothing to do, for example this week I had 9 meetings, he had one. He gets paid double what I do so that pisses me off, but the real problem is that since he has nothing to do he spends his time gossiping about people and planning things like basketball and volleyball games to go play during work time. He even planned a beard growing contest in all of his spare time. The thing is I am too busy to participate in everything he plans, or I just don't want to, but the other guys in the office love them and they go and hang out together. I am the "new guy" so it already is awkward and I can tell that I am quickly becoming the outcast. What am I to do about this? I try and be friendly and start conversations with him, and he is usually pretty nice, then goes and blows me off to do buddy things. My plan was that I was going to work at this place for a long time. If it is already like this with only four months, I don't know how I will survive.

What would you do?