Should i walk away even though i love him ?

I been dating this guy for about 4 months. Im having issues as to whether I should stay or end things with him. I feel like he always has excuses on why he's not spending time with me. He always says are work schedules get in the way too. I work overnight and he's a bus driver that works evening shift. But I feel like he dosent even try. He tells me to give him some time and I been waiting. He doesn't call me barely and when he does its late at night when he's outside. I question why he only can talk to me outside and he says he dosent like being on his phone in the house. Hes still married because he says his wife doesn't wanna sign the divorce papers. But my gut is telling me that his wife still lives with him and their kids. I asked him about it and says they don't live together and that he lives with his people. I recently found him talking to other females in his phone and we had a big argument about that. I love him and he said he loves me but im not happy. And im hurting.