problems letting go of 4 year old son?

I am 40 years old and already have 2 girls (19 & 20 yrs. old). Now have a 4 year old boy. I had tons of help with the girls my mother, my in-laws, & their father was always around. (not always a great dad, but was still there). Now my mom has passed away, my son has different grand parents, and his father is a truck driver that might be gone for months at a time. Even when he is home he is not a lot of help. I never get time for myself, because I don't want somebody else to keep him. My father takes him quite a bit,(for me to work) but he cries when I take him there. He likes to go to his other grandparents, but they always have an excuse why they can't take him. (this is even their first grandchild). He is speech delayed, so he wouldn't be able to let me know if something was wrong. I work evenings & getting a sitter is very hard. There is one girl that watches him sometimes and he loves her, but she won't watch him in our home. So when I get off he isn't bathed or ready for bed. (she doesn't watch him at home because I live in my sons dads house) and her parents don't want her here. Any way to the point I don't want to put him in daycare or anything like that. He is speech delayed and he gets frustrated. I understand a lot of things. He is in speech therapy and goes to pre-school. But I have to take him to speech therapy and he can only be picked up and dropped off at the same place every day. I don't know what to do. I am late to work a lot, have to wait for his bus. My co workers are getting angry. When he gets sick I have to stay home. I just want to be with him. I have been thinking of babysitting and quitting my job, what should I do?? I just feel like no one really wants him around but me!!