Why is he saying (I shouldn't have been)? I didn't understand.?

The author realized, in retrospect [looking back], that he shouldn't have been surprised that this person always remembered his name. The fact that he knew hundreds of people at the event, more people than anyone else, would indicate that meeting people and getting to know them, 'networking' on a grand scale, was the very essence of the man.

I read this and what came to my mind was, perhaps something happened after meeting him and had he known this info he would not have been ddelighted the guy remembered him. Retrospect meaning to look back...

At the time I was amazed.

But now I know more - looking back ( in retrospect) I actually should have expected it.

I would say that he was so impressed with the person in question that, while he was initially astonished that he remembered his/her name, in retrospect, he found him to be so impressive that it ought not to have been a surprise.

There was a later unflattering mention of the author by the candidate in a public forum, or his reputation was being used to further leverage the candidate without the writer's express support.