My special needs neighbor won't leave me alone! How do I not be mean about this?

My neighbor has special needs and has an aide to take care of him. They go out for walks every day and that's how we met. The first time I was about to leave for work and he asked me for my number. I didn't want to give it to him (I never give out my number) but I was also running late, so I used that excuse. The next day he saw me again and I was leaving to the mall with my boyfriend. He asked for my number again and I didn't want to be mean so my boyfriend wrote it down for him. Not even 20 minutes later, he called me to ask where I was, what I was doing, and when I'd be home. Because of his disability it's very hard to understand him, so this conversation took a while. When I got home he called me at least five times. I finally answered (10:00 at night) and of course couldn't understand him so his aide got on the phone. Apparently he wanted to come over. I'm not one for ANY kind of company and told the aide that we were settling down for the night and the calls stopped. Today he called 11 times and knocked three times. How do I make this stop without hurting his feelings? Shouldn't his aide be stopping him from calling so much and inviting himself over? I understand he's a bit slow but surely this isn't ok???