Better job offer....oh no!?

I've been submitting applications to several different jobs for a couple of months now. A week or so ago I got a call from a company I had turned in my application to only a couple days prior. I had my interview, and they called me and offered me a job. Today, I went in to finalize paperwork and the like. When I got home, I checked my messages and heard an offer for an interview from a MUCH better job (better benefits, likely better pay). I haven't started at the job I first accepted...and my question is, should I even bother interviewing with the better job? Should I leave it at "oh well, things like this happen," and start my first job? Or should I interview at the better job and decide what to do if they want to hire me? Would it completely wrong to give 2 weeks notice to a job I've been at for less than a month if they DO want to hire me???

HELP.