Help me, is my family really perverted or am I just paranoid?

Some background, my older brother did actually behave inappropriately towards me from about age 12 to 15. And when I was 7 an older boy (not related to me) took advantage of me. And I have an uncle who made inappropriate comments.

My problem, I am now verry uncomfortable being hugged, touched, or even too close to male family members. I am always wondering what is really in their minds when one hugs me or makes a joke or compliments me.

I most often am uncomfortable around my father. He has never done anything pervy to my knowledge but the things he does such as joking about it when a commercial for Victoria secret comes on, hugging me normally, poking or teasing me, opening my bedroom door slightly to talk to me or let my dog back in, even though he doesn't look it still feels like an invation of privacy, and worst today, he came up close behind me and poked me in the sides. It all seems innocent enough right? But I get freaked out anyway. He is a good person who would be disgusted to see a story on news about someone molesting etc. someone else, and yet I feel uncomfortable. Could my mind just be over reacting to men because of my past. I have little problem getting close to female family members, just the male ones I always feel worried about.