Why is it only "poor" neighborhoods that have drug gangs?

Thats not strictly true, some of the middle and upper classes are involved albeit by invisible means.

I live in Philadelphia, and wealthy people from the suburbs come into poor neighborhoods to buy their drugs on the illegal open air drug market. People with money would never tolerate the crime and violence that comes along with the drug trade in their own neighborhoods (or subdivisions, more likely) because they have the money, power, and influence to prevent it. People who live in poor neighborhoods do not have the means to kick out the drug dealers. Many of these neighborhoods have very low population density, property vacancies that fuel the drug market, and not enough policing since there is not a dense enough population to provide the taxes necessary for more police. It's a vicious circle. In the drug-infested neighborhood where I did a project for grad school, the neighborhood residents did not want the drug dealers there, but they had almost no control of the situation because of their lack of resources.