How to deal with this person?

Intro: (skip if you don't care- but it might help answer)

When things are going well for me, I can deal with anything and anyone (at least I feel like I can). I can turn an angry man into an agreeable fellow, I can persuade a resounding no into a 'OK' or I can make anyone, no matter who they are, feel better when they are around me. I'm also very honest, I don't feign interest, I say what I think, and generally keep my integrity. All great right? However there are some things i cannot do so well, and when these build up, I can slowly turn into the depressed, antisocial, mood killer version of myself. So I need to deal with this in the best way!

Question:

So here's the problem, someone I work with. He's a very sociable character and pretty intelligent. He can talk to almost anyone, I imagine he was once a confident guy, but now he's definitely picked up insecurities. We also have some history of going out together. He is a really nice guy. But he hurts my head! For short periods of time I can tolerate (I'll stay sane and leave his company with furrowed eyebrows wondering why he hurts my head so much) For long periods of time I can get really impatient. Only thing is I feel like if only he had more confidence in himself he could be a really cool guy. But he really drains my vibe! How do I deal with this guy? Do I try fix it or do I drop him? I always see myself as a conscientious kind of guy, but I don't know what the conscientious thing to do would be.