How do you tell someone you don't want to have sex?

I have an old friend with whom I've never been physically intimate, although it almost happened once or twice yeeears ago. Lately, I've moved closer to where he lives, and would like to hang out with him; however, he keeps making references to us hooking up whenever I come to visit (and no, he's not hinting, he's talking like we already agreed to it)... I guess what I want to ask, is how do you tell someone: "I don't want to have sex with you." And yes, it's an attraction issue, I am not physically attracted to him.