Feminine troubles with soap?

I know this is an awkward subject but I've always wondered if other women deal with this.

I can't use the same soap as my family (as in from the same soap dish) because I'm prone to breakouts down there. I get lumps that go away after a short time, so I've learned to just keep my own soap in traveling dish. This should be a done solution but it's not. I suspect when they run out of soap and don't want to retrieve their a new bar of their own they reach for mine....and if I suspect my soap has been used by others (or dropped on the floor) I automatically have to throw it away.

Do other women have this problem with soap and their lady parts? I mean....is it normal?! I didn't used to break out from the communal soap until maybe 4 or 5 years ago.