At what point in your relationship did you tell your boyfriend/girlfriend you loved them?

I have known my current boyfriend for a year and a half and although we have been on and off for almost that whole time, we basically started to actually really see each other (as in I met his parents) in March. However, we only officially declared that we were together to each other in probably May (even though knew friends, family, stayed at his house every night). He asked me out when he was incredibly drunk over the phone and said that he was tired of not knowing what we were and he loved me. But he has never said it ever since and he does get quite emotional when he is drunk so I'm not sure what it means. Lately we fight a lot and he has said to me, if it was any other girl he wouldn't put up with this, and I feel like this is because their is so much ambiguity around our relationship. I think for him he doesn't trust me because I spent a few night with one of his friends (before I knew who he was and was unaware that they were friends) and for me because there was a rumour debacle where he "apparently" spoke ill of me. However, generally we get along incredibly well. Would it be appropriate to say I love you or even have a conversation about it or does it seem to messy, because I really do feel that way about him but I don't know if it will make matters worse.