Should I avoid buying a car with all-wheel drive?

I'm looking to buy a late-model used Honda CR-V (ideally, 2-4 years old, and under 35,000 miles). I don't really need all-wheel drive. I live in Seattle, where we don't get much snow. I hardly ever do any mountain driving in the winter. I don't plan to go off-road. I will mostly be driving it around town--commuting to work, shopping, etc. I mainly like the CR-V because of the size, appearance, and reputation, and because it rides higher than a lot of small cars (I have bad knees--that makes it easier to get in and out of). I intend to drive the car I buy for a long, long time (I got 17 years out of my last Honda).

But...there seem to be a LOT more all-wheel drive models out there on the used car market than front-wheel drive ones. I have only some vague ideas about all-wheel drive--that it lowers gas mileage, and that it's just something else to break down. But if I rule it out, my choices of available cars that meet my other criteria will get a lot narrower.

SO...can somebody help me better understand all-wheel drive? Are my ideas about it outdated and off-base? Is it going to be a liability in my situation?