Why people don't like to physics as there college major?

Typically because they're failing at it. Hard to get enthused over something when you're not good at it.

But to a point, there are lots of people who really like physics as their major. And they graduate with one or more degrees in it.

People have the mistaken impression that "physics is hard". Most of the difficulty in early physics courses stems not from the physics being too difficult, but from the students not being properly prepared in their mathematics. Often students are learning physics and calculus at the same time. I learned much more mathematics from my physics classes than I did my math.

Also, physics is not a subject where you can simply read a bunch of facts and regurgitate them later. It requires critical thinking and understanding.

Finally, to do well in a subject one has to have an interest in it. Not everyone finds physics to be interesting....hard to believe, but its true.