Need help with Google Voice...?

I had a prank call to my wife's cell phone on Friday night that I was able to trace the other night. I called the number from my Google number so not to display my "real number. Well the crazy drunk guy did some type of back trace and showed up at a house I lived at 20 years ago. How did this happen? How did he link that address to that number?

As when I try to back trace the number I find no address as it is a free Google voice number.

Long story short the 'Drunk Crazy Guy' is in jail as he assaulted the person that lives there now. And when you think you have seen and heard it all. Bam something NEW!!! lol

Thanks for the help. :-)