Derive the rotation matrix R that represents the rotation about a line by an angle A?

*edited after reflection

This can be tackled by by calculating the product of 5 separate rotation matrices based on angles. Two to align one axis with L, one to effect a plane rotation about L and then two inverses to restore original axes. However, calculating the elements of the overall rotation matrix is not exciting and result seems without form. This way uses vectors and gives an explicit expression for R.

WLOG we can set r₀=0 and do translation at end. Let n be a unit vector such that d•n=0

Establish orthonormal axes OX,OY,OZ where OZ=d, OX=n and OY=dXn=m

Set n=k(d₂,−d₁,0), m=k( d₁d₃, d₂d₃,−k⁻² ) with k=1/√(1−d₃²) but use later … (i)

The vector coords of P(=(x,y,z)=r) in XYZ system are ( n•r, m•r, d•r )

After a RH rotation A of P in the XY plane, these become

( (n•r)cosA−(m•r)sinA, (n•r)sinA+(m•r)cosA, d•r )

In original coords this rotated point is r’ where

r’ = (x’,y’,z’) = {(n•r)cosA−(m•r)sinA}n + {(n•r)sinA+(m•r)cosA}m + {d•r}d

To transform to matrix form Rr use (p•r)q = (pᵀr)q = q(pᵀr) = (qpᵀ)r on each term

This gives R = (nnᵀ+mmᵀ)cosA + (−nmᵀ+mnᵀ)sinA + ddᵀ

Using (i) the last row of R is {−d₃d₁−d₃d₂,1−d₃² }cosA + {−d₂,d₁,0}sinA + d₃dᵀ

Because R is unique and X or Y axis could have been aligned with L instead, R should be independent of n like the row above and I claim that because d•n=0 so then

nnᵀ+mmᵀ = { {1−d₁²,−d₁d₂,−d₁d₃}, {−d₂d₁,1−d₂²,−d₂d₃}, {−d₃d₁,−d₃d₂,1−d₃²} } = I−ddᵀ

Likewise −nmᵀ+mnᵀ = { {0,−d₃,d₂}, {d₃,0,−d₁}, {−d₂,d₁,0} } and call this matrix K

So R = (I−ddᵀ)cosA + KsinA + ddᵀ

This result gives the right answers for d=i,j or k.

With this matrix the coordinates of the rotated point are r₀ + R(r−r₀)