how many people blowing at the same time...?

You probably couldn't concentrate enough people into a single location to blow down a building. Imagine getting the entire population of the Earth (7 billion+ people). They would each require some space of their own, and soon they would dwarf the building you're trying to blow down.

Ok just for fun.

First realize that wind covers the entire surface area of the building, now air comes out of your mouth which may be, 3 percent the surface area of your front side so a person would be only using 3 percent of the same surface the wind would. And a person in front of someone else would block that air from the person behind for the most part. And in no way can a person blow as hard as a decent breeze so I would argue no amount of people could blow down a building. In fact you get that many people in front of a building, even blowing, and the building would feel less force than even a gentle breeze.

Well, no amount of people can do it.

You see, it's not the amount of wind, but the speed and amount that blows things over. And while you can have a lot of people blowing they can only blow so hard; and that's not hard enough.

Some structural damage begins with a "strong gale, 47 to 54 mph". [See source.] But less than that, structures, your building, will not be damaged at all. And unless you know some blowhards who can blow at over 50 mph, no amount of people can blow down a building.