Why is"He wasnt born with a silver spoon in his mouth" rude?

It can be suggesting that the person being spoken to was, the speaker rudely implying the person he or she is speaking to is an entitled snob who's had everything handed to them on a silver platter, i.e., never earned anything and is undeserving. That's a pretty rude judgment to voice, especially if untrue but also even if there is truth to it.

How is not coming from a wealthy family rude?

I guess it means that he’s not pristine 

You must be talking to a rich person. Rich people gets offended by a lot of things when comparing to the poor.

Yeah, I mean, it's much more polite than just saying he's poor.