Opinion on changing my major?

I'm a student in a California Community College, about halfway through my General Education. I've been pursuing transfer to Cal State San Bernardino for a Bachelor's in Business Administration with a concentration in Accounting. Up until tonight, I've been fairly certain about this major. While Business Administration degrees can be dime a dozen, they're generally a good fall back plan. However, that's admittedly exactly what this is; I more or less decided to pursue this simply due to lack of any other known interest.

After a discussion with my boyfriend about my education, he brought up this fact (I never told him, it's just obvious), and asked suggested I change to something else. More specifically, he suggested Physics. Science has never really been my strong point (hated Biology), but math on the other hand is an absolute passion of mine. I also love learning constantly (nerd, I know), and having a future that gives me that privilege would be a dream, but I'm not sure if that's something a Physics degree can offer me.

What I'm getting at is I want the opinions of other on this, namely because I have to do a near 180 with future classes to accommodate a change. Would this be really be something I could really enjoy, and should pursue instead?