Should I Not Attend Family Parties Because of My Dad?

Although not properly diagnosed as this, I (along with many others) believe that my dad is a narcissist. To add, he's been verbally and emotionally abusive toward me, and I once threatened him with a restraining order because of his harassment. I haven't had contact with him for a while, which relaxes me.

However, I am only close to his side of the family (as opposed to my mother's), and my family likes to throw holidays feasts and reunions often. Occasionally, my dad will fly down to my state (for we live apart) to participate.

I enjoy seeing my family and eating with them, but I've yet to tell anyone aside from my mom that I fear seeing my dad again. I don't want to stir drama, nor do I want to cause a rift between me and my relatives by telling them that it's either me or him. I hear he will be flying out this winter for Christmas. I want to attend the parties like usual, but I won't be able to handle facing my dad.

Should I go, regardless of if my dad comes? Or do I decline the invitation, and risk upsetting people?