Can I close out my traditional and Roth IRA accounts to use for down payment?

I have a traditional IRA and a Roth IRA with relatively low balances that I would like to withdraw money from to use for a down payment on my first house. I know that it's a qualified withdrawal so no penalty, but is there a penalty if I close them out completely and leave zero balances? The funds are from previous employers and have just been sitting idle (or declining and rebounding in the economy, I guess) and I have a newer, higher balance Roth that I currently contribute to. Is it ok to just close the old ones out??