Should I allow my 6 year old son's father to see him?

Myself and my son's father split up before he was born however I wanted my son to know his father. My son did'nt bond with his father as a baby because his father did'nt show any interest. His father lives with his mother my son's grandmother whom my son has grown very close to as she is the one who has looked after him whenever he has gone to visit. My son would go every weekend, but his father was never there to greet him or to say goodbye when I picked him up. In the summer of 2006 shortly after my son's 3rd Birthday I got a phonecall at work from my son's grandmother who was having Jack at the weekend, she told me his father was being nasty to him taking his toys off of him because he would'nt play with him. I was horrified and called my mum immediately to go and collect him. I stopped my son going atall after that and told him he could'nt see him. He told me if he could'nt see him then neither could his Grandmother who my son absolutely adores. In the new year of 2007, after communication with his gandmother who assured me that his father really missed him and wanted to see him we arranged for my son to start visiting again just for the day instead of staying over. This led to him enjoying himself and looking forward to seeing his nanny again, and then staying over. His father was never there when i dropped him off or picked him up and I noticed that when he was they did'nt speak, he did'nt even say hello to his little boy or goodbye, didnt even aknowledge that he was there which I grew to fnd quite alarming. Is this what I want my son to grow up thinking a father should be like?? NO WAY !! So I decided in my son's best interest that he did'nt go anymore and his Grandmother could see him whenever she liked. My son is a very quite and shy boy to begin with and was very quite in the pre-school he attended untill one of the teachers pointed out that he had come out of his shell greatly since not seeing his father. Jack started going again on and off mainly starting seeing at Birthdays and Christmas would start them seeing each other again. The next issue happened when his father demanded that I drop my son off to him and not his mother his mother must'nt have anything to do with it. I thought this was rediculous as my son adored her nd his father was clearly jealous of their relationship. Very immature, I told him this was not going to happen and so he threatened me with court. I told it was rediculous to have court involved I just wanted grandmother there when he was there for peace of mind. He would'nt have it and so I stopped my son going. I feel that his father is not thinking of his sons best interests and more that he is his father and this is how it goes. He does work and pays maintenance, which he stopped everytime I stopped him seeing him. He is just very immature and I want the best for my son and don't trust his fathers character to be responsible enough to take care of him. After his father got intouch with a solicitor they contacted a mediator who instead of going to court tries to come to an agreement with both of us and settel out of court. We went to a mediator in June 2009 and came to the conclusion that my son would go everyother weekend for a sleepover. His father spoke of wanting to be more of a father and I spoke of my mistrust of his past actions and there lack of a relationship. We agreed he could have another chance to build a bond with him, which would'nt be hard if he tried. October 2009 just over 4 months later and my son say's to my mother 'my dad slapped my face'. My son is a lovely, very polite, well behaved little boy he would never deserve a slap and definately not in the face. This had obviously worried him and was on his mind, when my mother asked him why he told her for no reason and when she asked him wht he did hesaid he went and told his Nanny. Who said she would have a word with him. When my mum told me I txt his dad and asked ' Did you slap him in the face, and if so, why?' His reply was 'we play fight sometimes, other then that I dont know what your on about' the following weekend he asked if he was coming or was i stopping it again, when i asked my son if he wanted to go and see him he said no. so i told him he didnt want to go. and got no reply...I went to the mediator to ell tem about the slap and they said it would have to be delat wit seperately with a solicitor because the mediation is a joint thing between both of us.

I am really confused and worried about my boy, he is struggeling at school, has problems with socialising and communication. And i am petrified it because of his fathers influence.

Please somebody tell me what need to do next ??