Is it just autistic people getting left out or is it just anyone gets left out?

Welcome to the wonderful world of being an Autistic adult. Yeah, you're going to get left out and treated like sh*t because of your Autism. I know, because I've dealt with the same crap for years- even decades. I didn't even find out I was on the spectrum until I was past 50, in fact. I just suffered a lot of abuse and mistreatment, without ever understanding why.

US society is one that discriminates horrendously against anyone who doesn't meet whatever the socially accepted standard of "normal" or "cool" happens to be. And that standard can change rapidly- so fast that it's impossible for most of us to keep up with it. In the minds of far too many so called "neurotypical" people, even being slightly "different" makes one "unacceptable".  That's sad, it really is, but it's a reflection of how screwed up the US is, especially when it comes to attitudes towards not only Autism, but mental illness and other developmental disabilities too.

Yes, even us NTs get left out sometimes.  It can happen for various reasons. As an example, you might meet people at work with whom you work really well as a team.  Everyone is polite to each other and often do each other small favors to make the day pleasanter or the work easier, which of course also makes the day pleasanter.  However, you might not "click" on a friendship level.  Then you find out that several of your co-workers get together after work or go away together on the weekend.  I'm not going to lie, finding that out hurts at first and it can feel like an attack, but if you take a step back (this can take a few weeks) you realise that you just don't fit together as friends but they way you work together well and the way you like each other at work is real.  Of course the worst thing is when you introduce a friend to another friend and they like each other more than they like you and start to phase you out.  That happens too, and yes, it does hurt.