What would happen if a person on Social Security didn't file income tax?

Their system would catch up with you and eventually, deduction (including penalties)  would be made from your SS check.  There's no need for the IRS to "go after" someone like that - SS would generate a 1099 and IRS gets that info - totally automatic. 

The thing is, when on SS, that few dollars makes more than just the few dollars taxable.  

The IRS would send a bill for the missing tax plus two penalties for late filing and late payment.

It's very easy to "go after" people with simple incomes who failed to file. The IRS gets a copy of your 1099 and W2 forms; their computers automatically know whether you owe money. It takes a lot more work to go after people who claim a bunch of ridiculous deductions and who disguise gifts to their children as business expenses. Auditing those accounts would bring in a lot more money (and discourage others from lying to the IRS), but since those people run the government they've fired all the auditors.

For a couple of bucks IRS has bigger fish to fry.

"Go after"?  Th person would get  a computer generated  letter billing them for the 'few dollars' they owe.

Generally, the IRS wouldn't be interested. They wouldn't waste their resources tracking someone down over a few dollars.

They're more focused on more extreme cases of tax evasion.