show the set Aut(G) of automorphisms of G forms a group?

Let Aut(G) be the set of all automorphisms φ: G --> G. In order to show that this is a group under the operation of composition, we must verify:

(1) Is the set is closed under composition? Yes! If you are given isomorphisms φ, ψ: G --> G, then it is not too tough to show that ψ∘φ and φ∘ψ are isomorphisms. I can expand on this in more detail if you like, but you have probably seen a proof before that a composition of bijective functions is bijective. If a and b are elements of the group, ψ∘φ(ab) = ψ(φ(ab)) = ψ(φ(a)φ(b)), because φ is an isomorphism. Since ψ is also an isomorphism, ψ(φ(a)φ(b)) = ψ∘φ(a)ψ∘φ(b), so the composition ψ∘φ preserves products. Thus, ψ∘φ is an isomorphism if ψ and φ are.

(2) Is the set associative? Yes! All you need to do is show that, for any three isomorphisms φ, ψ and ξ, φ∘(ψ∘ξ) = (φ∘ψ)∘ξ. To do that, just show that for each x in G, φ∘(ψ∘ξ)(x) = (φ∘ψ)∘ξ(x) = φ(ψ(ξ(x))). It's just pushing around definitions.

(3) Does the set contain an identity element? Yes! Let the identity automorphism e: G --> G be the map e(x) = x. Clearly, e∘φ = φ∘e = φ.

(4) Does each element of the set have an inverse under ∘? Yes! Since each isomorphism φ: G --> G is bijective, there is a well-defined inverse map φ^(-1): G --> G. You may have already seen a proof that the inverse of an isomorphism is an isomorphism. If not, it isn't too difficult to prove: I'll leave it to you, but I can expand on it if you need me to. Further, the composition φ^(-1) ∘ φ = φ ∘ φ^(-1) = e.

Since Aut(G) satisfies all the group axioms, it forms a group under ∘, as needed.