I feel like I'm second best?

I always get frustrated with my boyfriend and I feel like he'd always put his friend before me. On numerous occasions he's been at my house with me, his mate has text him and he makes me drive him home immediately so he can go somewhere with his mate. He spends loads of time with him, I don't mind but I hate it when he drops me for him. Today his mate text to say he's picking him up at 3:30. He made me drive him back at 1:30 incase his friend waa early. As apparently his mate will be annoyed with him if he turns up and he's not there. But in my opinion, if his mate says 3:30 and turns up early, he shouldn't necessarily expect him to be there as it's not the time they agreed on. It kind of upsets me but he doesn't seem to understand. They see each other every day, so I'm not stopping them seeing each other. But am I fair to be a bit annoyed? I also think his friend is a bad influence. He takes drugs, shop lifts and who knows what else! Personally I don't think he's a good friend and his friend always thinks he should come first over me. I feel as though he almost uses my boyfriend, but my bf thinks he's like God.

Am I right to be annoyed? How do I tell him its annoying? I don't want him to think I'm stopping him seeing his friend.