how can i now not be fearfull of men alone?

right cut a long story short ..im marrid ,i am a cleaner and have been cleanig for a man in his 60's for 6 years i have always felt comfortable cleanig for this man and just got on with the job.but something has been brewing for the last few months where i have been more assertive around him,i was hoping it was me just being paranoid or something .but he has been very suggestfull and inviting me round for dinner ,not inviting my husband just me,and i kept saying no and changing the subject.well thursday was the last straw he said he expects me at nine pm .leave your daugher in the back bedroom bring your toothbrush and we will have some wine,,i said no im busy and no thank you !!..he slammed his cup on the table in anger and that was enough for me ..obviously im never going back there..but that was a wake up call ,i really didnt expect that and it confirms what his motives have been all along.i feel so scared and vunerable,and i feel i dont want to b with men alone apart from my husband dad and brother..if i feel like this with just a wake up call how would i of felt if next week he did grab me?..i dont want to be a drama queen but that was absolutly un called for ..you dont go around making that kind of talk with marrid women.even worse he has a partner!!..so i dont undersatnd why he was like that with me..how can i get rid of this fear ?..and he only lives around the corner .oh i just feel really odd now ,please help!!