Should I follow my dreams, or settle for security?

I'm 20 years old and withdrew from university after my sophomore year because I was stressed with school and needed a break, I landed a job as an IT Support guy with a decent company and I somewhat enjoy it, work isn't awful pay is okay for someone my age with no expenses and much longer experience can lead to great opportunities employment wise in the future. But the other side of me wants to follow my dreams, no matter how slim they seem. The thing I've noticed while working at this job is the other people I work with are so passionate about it and all they do inside and outside of work is IT things, nothing else, so I thought to myself maybe if I did what I truly loved more than anything I can be as happy as they are, the only problem is the only thing I love more than anything is the game of Baseball, I wasn't physically gifted enough to play the game, being 5"3 and having numerous heart defects that prevent me from serious physical activity, but since I was 5 years old I ate, slept, and breathed baseball. It's the only thing I turn to and I feel like it's the only thing I can do in my life that will satisfy me and make me truly proud of my life. What I dream of is being a MLB GM but I understand there are only 30 of them so I should shoot for something a bit more realistic, like a scout or something along those lines but baseball is all I know. All in all my family tells me that giving up this gig to follow a shot in the dark dream is ridiculous and I'm torn.