Would this writing technique bother you?

In the novel I'm currently writing there are several characters with events in their lives that are very important to the main book. The instances happened in the past, and many are very complicated, so making them an actual part of the main story wouldn't work (at least not to the extent I would need it to). My solution was to write a chapter for each character, the chapter being a short story detailing the event. These chapters have nothing much to do with the action of the book except that they affect the character's later actions. They appear directly after each character is introduced.

My friends who have read the first draft are divided as to whether they like this style. Some do, and say that it's a good way to bring a character into play, it's an interesting aside to the main plot, and that these chapters are positioned in places where they don't interrupt the flow of the story. Others don't like it because they're irritated by the idea of a short story in the middle of a novel and say I spend too much time trying to establish an entire character in one place.

I personally like the way it turned out, but there are enough people that hate it to make me reevaluate. I can see this from both sides, and I'm sort of stumped as to what to do. Would this bother you as a reader? If so, how would you suggest I change it to make it less of an annoyance and still be able to give the necessary information for each character?

(And tell me if you need anything cleared up; I'm not sure I've done a very good job explaining this.)