How would I let them know...?

I have this thing with being touched, I absolutely hate it, and freak out. Accidental touches don't bother me, I barely register them. When its done on purpose, I freak, and say (almost yell) "Don't touch me!"

The weird part is, some people can touch me, and I have no problems with it, it doesn't freak me out, other people touch me, and I freak, it makes my skin crawl. There is no gender preference, there is no trust thing (that I know of) as some people I trust with my life, I can't stand them touching me, and some that I don't trust, them touching me doesn't bother me.

But there's this one guy, I have a crush on him, and when he touches me, it doesn't bother me, and I don't know why, but I'd like to let him know that just because I seem upset with him after he touches me, that it didn't bother me, and I kind of liked it.

By touching I don't mean fondling or anything, I mean like a tap on the shoulder, or trying to tickle me, or a hand on the back or shoulder letting me know they are there or that they need me to move.

As far as I can tell, the crush doesn't have a thing to do with my being touched issue, as I've had crushes on guys from afar and then when I got to know them and they'd touch me, I would freak.