What can cause a spiral fracture in the femur of a 5 week old infant?

On Tuesday, around 4 am, my husband was getting our 5 week old son out of his swing. It's a very normal, nondescript swing. When getting him out, my husband had one hand supporting his upper back/head, and the lower he put under our son's bottom. Baby squirmed my husband ended up with his leg in his hand instead of his rump. He heard a pop, so we immediately went to the hospital. X-rays showed a spiral fracture of his femur. We were shocked. My husband did not apply near enough pressure to break a bone. Of course, CPS does not agree. Even though there is no evidence of any abuse to any of our children, not so much as a bruise, we are being threatened with removal of our children and possible jail time. So my question is this: Does anyone know what possible bone problems could be behind it? He is growing extremely fast. Can rapid growth attribute to fractures? We know my husband did nothing wrong, and are worried that there's something wrong with our son to cause him to be injured so easily. Any advice on infant bone health would be most appreciated. We're in a pretty bad position.