Why do some people think that a rooming house should only house people on parole, or crack pots, or prostitutes?

I'm living in a rooming house, its not comfortable, in fact I hate it a lot of the time, but I was just starting out on my own, with a a new job and hadn't yet gained my hours like I have now. I understood at the time of moving in, that this was a roof over my head while I worked my way upward and out of here.I get a lot of strange looks from the people who live here because sometimes I leave for work during the day and don't return until 11 pm. Why do they concern themselves with my whereabouts? I the girls who live here use guys to get their rent paid, they have come to my door offering "services" they get mad when I say no and don't ever come knocking on my door for something like that....they want to know how I get my rent paid, is it my right to tell them its none of their business, or do I say that I ork a real job, earn honest money and my rent is paid for in an honest and clean way.