what could happen with all the stuff removed from underground?

The earth's crust takes up less than 1% of the Earth's entire volume. The continental crust (the crust occupied by our land masses) is anywhere from 20 to 30 miles deep. The deepest mine in the world is about 2 and 1/2 miles deep. That means we would have to dig 10 times deeper to even come close to the mantle/crust boundary. So, you don't have to worry anytime soon about the earth collapsing on itself because of mining.

That said, there are still dangers from mining and drilling that we should always be mindful of. While a massive, earth scale collapse is very very unlikely, local cave-ins do occur. But even this is a relatively rare occurence, even with the 1/2 million abandoned mines we have in this country. We DO have to work about excursions into our groundwater of chemicals and gases that are by-products of mining. This can be a BIG problem and unfortunately, there are some massive abandoned mining operations (especially in the western US and probably elsewher in the world) that have yet to be cleaned up. These mines were used at a time when the EPA wasn't around to regulate the industry. These are really what should concern us at this point.