What is the difference between public and private information?

I understand it is legal for a private citizen to collect public information on a neighbor, friend, potential date, fellow church member, etc. Is it illegal to collect private information, especially without their knowlege or consent? What is the difference between the two - public and private? If someone posts personal information on the internet, does it become public information, for anyone to use? When does collection of personal information become stalking or some other crime? Or is it just what they do with that collected information that counts? (Like accusing you of a crime or extorting or blackmailing you?) Is the act of following a person down the street with a cellphone video camera running, or taping conversations without the knowledge or consent of the party considered stalking? Or is it okay to do such things on a public street or at a mall or at school, versus on private property?