I am invited but I don't want to go, any advice?

Hello, my boss of the company where I work at invited us for Iftar to his villa ( Iftar is the meal eaten by Muslims after sunset during Ramadan), and we the employees are more than 30 I guess, but the problem I said I will come on 6 July, but I was confused what should I say, but I don't want to go, I hate being invited, to be honest I hate being surrounded by large groups and I don't even like to talk too much, I'm a quite and calm person, maybe because I'm a graphic designer, I spend my time designing alone, I'm afraid they may say something about me like he doesn't like to sit with us, he is isolated etc...

What should I do ?

Yeah and plus there are some employees that I don't like