Why do guys my age have a hard time losing weight?

I'm a 44yr old male, athletic for most of my life. Last few years I've been overweight and going from 275 down to 250 and now back up again. Recently since I changed my diet, I've gained weight and can not shed it. I stopped eating pork, beef and chicken. I eat very light breakfast; 1 cup of coffee, a banana and cup of yogurt, usually a salad or sandwich for lunch, an afternoon snack (something healthy) and then at dinner usually some fish and veggie plate. I do have a cocktail or two at night, but not more than that.

I go to 1 hour boot camp 3 days a week, which I end up sweating profusely and always feel a lack of spring or energy. I am also an avid cyclist and when I'm not going to boot camp I regularly ride a minimum of 40 miles per week. I am a strong cyclist and my legs are conditioned, but when it comes to running, weights, or push-ups (boot camp stuff) I am weak. I currently am 6'4 and weigh around 300lbs. I have an office job and sit on my butt most of the day.

I'm having second thoughts about my diet. I changed because I was disturbed after watching a movie called; "Food Inc." and no longer wanted to participate eating animals (not including fish), but because of all the stuff going on now, I'm considering going back to eating organic only chicken, but no red meat.

I thought maybe I was eating too much carbs, but when I went to eating just salads with tofu at lunch time, I feel low energy for my boot camp or cycling. My carnivore friends say that very few good athletes can perform without eating meat for their protein needs.

I know my body pretty good, so I am just dumbfounded how I am putting on all this weight now, and attribute it to the change in my diet. But then again, it also could be attributed to my age, "Low T", or any number of reasons guys my age start to put on the poundage. I'm literally consumed by my weight gain constantly thinking about it. I would never have imagined, turning out this way, since I grew up so skinny. I am really disgusted by it to be honest and want to change badly. What can I do?