I Don't Know What to Do?

I work with a man who've I've developed an attraction towards, and it's been returned by him as well.

Sadly, I've heard these rumors from a couple people that this man isn't the greatest. Yet I've been spending a lot of time with him and have been getting to know him and he is nothing like how these rumors have led on to believe. He's been very nice and sweet to me. So either those rumors aren't true (even though I've heard them from two different work places), or they were true at one time but now he's changed, or he's just a really good liar to me. Though I don't understand why he would even try lying to me when he knows very well that he can't get anything out of me.

I feel really torn. I've started to like him, and I'm scared that the rumors might be true and that if he and I start dating I'll only end up getting hurt.

What should I do? Do you think this is a subject worth talking about with him directly?

Thanks.