I hate my new job (advice needed)?

I took on a second job to earn some extra money.

I was basically hired on the spot after a short chat with the boss. It involves a lot of heavy lifting and I am on my feet for most of the day, which is pretty tiring to say the least.

My co-workers think that I am strange/crazy/snobby/stupid/etc. because I don't talk to them much or make stupid mistakes because of my social awkwardness... they think I'm "just shy" but the truth is that I have pretty severe social anxiety, something I have been working on recently, and my first experience with them was listening to them talk **** about our fellow co-workers.

The pathetic thing is that all of these women are at least 25 years older than me.. you'd think they'd have learned by now to have some compassion and stop being so judgemental.. I know I learned that as a child, for f$#% sake.

Anyway, it's not that I am "afraid" to talk to them, though I do get anxious, but I simply have nothing to say, and don't find much value in small talk.

I basically come home and I cry or want to get very drunk to deal with the stress that this job is causing.. I've only been there 2 days! Lol.

My question (finally :D) is, if I've worked there only 2-3 days, do I still need to give notice that I am quitting, or can I just thank her for the opportunity and tell her that the job is not for me..?

Any other job that I hated this much and didn't really need, I'd probably just never go back and perhaps call to let them know that I wouldn't be back.. however, I was referred by a friend that actually happens to like it there, and I don't want to screw it up for him.

Anyone else I've talked to has basically said to just quit since I hate it so much, while others have said to stick it out a little longer and see what happens.

Thanks for any advice you can give :D