What should I expect after a first date? Should I throw in the towel now?

He lives in New Orleans. I live in GA. We've been talking on the phone for almost 6 weeks & we've talked basically everyday for the past 4 weeks. He's in GA for the holidays & finally it's our first date. We had a good time-brunch, movies, hanging out a bit. He called on his way home to let me know that he was okay but didn't call when he made it home. Yet, he called first thing the next morning. We had a brief conversation & I ended it b/c I had to run errands. I called him back later in the day. Left a message on his voicemail. A couple of hours later he called me to say that he was going out but he wanted to "touch bases" so I wouldn't be mad. I was cool. We're not a couple. I decided to call him back for a brief conversation b/c it was still early (around 10pm). He said that he was getting dressed & would call me when he got in the car. He didn't call. I knew from the start that he hung out & was sociable, but I feel rejected. Should I give him his walking papers NOW?