What are my rights after being fired from church Job in VA?

I've just been fired from my church job. What are my rights in terms of communication with church friends?

My former boss (pastor) said that upon being fired, if I or my wife were to have any contact with any of the people with whom we'd made friends (regardless of whom contacted whom) then he (pastor & church) would file a lawsuit against me (no regards to Paul's teaching in 1 Corinthians 6).

Can he really file a lawsuit against my wife & me if we have any contact w/ friends from church? It was a big church...what if we see them in the mall or store, or park? We even bought insurance through a church member. Aren't all ties of control over me severed when I'm no longer employed? I didn't sign anything in agreement to this, and I wasn't fired due to any sort of legal issues or misconduct.

State: Virginia, USA

Please include a link in you answer as to where I may learn more.