Question about sign language, deaf culture and courtesy..?

I have friends who are deaf that I hang out with sometimes. Now, I know enough sign language to get by, but sometimes, they sign so fast that I can't understand everything, and I get lost. Now, some of these deaf friends are able to vocalize what they are saying (it's not crystal clear speech, but it's good enough that it helps a lot) but the thing is when they sign to each other they do not use their voices, which is fine. The thing is, when I'm with them, I have told them that sometimes they sign too fast and have asked if they could please use their voices along with the signing (which is easy for them because they do not use ASL, they use signed English). They very rarely do this for me. And I don't feel right in asking them every 2 minutes to stop and repeat what they signed, or ask them "what did you just say?" because this would happen a LOT...But they have complained that when I am verbally speaking with someone else in their presence (whomever - it doesn't matter if that person knows how to sign or not) they complain that I sometimes do NOT sign what I say. Now here is my issue...I know it is courtesy to sign what we say in the presence of deaf people (whether they are part of our conversation in the first place or not), in case they wish to join in our conversation. But I think it is the same courtesy the other way...so shouldn't the deaf (if vocalization is possible) use both their voice and sign at the same time if there is someone like me (who is not fluent in sign) around them? And I am only talking in a social setting...at a bar, party, whatever. Shouldn't we all take care to make sure everyone feels included? I don't expect the deaf people who cannot verbalize to do so, but there are some deaf people who CAN verbalize and sign at the same time.

What do you guys think? I am particularly interested in the opinions of those involved in the deaf culture.