A tetrahedron has edge lengths 4,4,4,4,4,x. What should x be to maximize the volume?

That is an interesting question. The first instinct is to say the volume will be maximized when x = 4. But I'm sure that the math will work out differently.

Essentially we have two equilateral triangle faces hinged on one edge with an edge of length x between them.

I'm going to focus on half the tetrahedron, so I'm imagining an equilateral triangle hinged to a flat surface. The height of the free vertex above the surface is x/2. That forms a pyramid of height x/2.

There's a right triangle formed, with the hypotenuse equal to 2√3 and one leg equal to x/2. So the other leg is √[(2√3)² - (x/2)²]

= √(12 - x²/4)

The triangle that forms the base of this pyramid therefore has one edge of 4 and a height of √(12 - x²/4). So the area of the base is 2√(12 - x²/4)

V = (area of base)/3 * x/2

V = (2/3)√(12 - x²/4) * x/2

Taking the derivative:

V' = (24 - x²) / (3√(48 - x²))

Setting that to zero we get:

x² = 24

x = √24

x = 2√6 ≈ 4.9

Answer:

2√6

Statement: The volume V of every tetrahedron is equal to

V = ⅔ S₁S₂ sin(θ) / e,

where S₁ and S₂ are the areas of 2 faces of the tetrahedron, θ - the dihedral angle between them, e - the length of their common edge.

Proof: Take the face with area S₁ as a base. Then 2S₂ = e*a, where a is the altitude (apothem) of the lateral face (area S₂) from the apex to the common edge (length e). Next, the right triangle with legs the altitude h of the tetrahedron (from the apex to the base), the projection of the aforementioned apothem and hypotenuse the apothem itself yields

h/a = sin(θ), or h = a sin(θ) = (2S₂/e) sin(θ), and

V = ⅓ S₁h = ⅔ S₁S₂ sin(θ) / e as required.

According the statement above, if a tetrahedron has 5 edges with equal length (e.g. 4), its volume is maximal if the dihedral angle between the two equilateral faces (the common edge like a hinge) is right:

sin(θ) = 1

(for example if 5 edges have unit length, the volume doesn't exceed 1/8)

This immediately yields a right isosceles triangle with legs both apothems (length 2√3) and the required 6th edge 2√3√2 = 2√6 as Puzzling has determined in his excellent answer above.

P.S. The plane geometry analog of the above statement is well-known: the area of a triangle is half of the product of lengths of 2 sides and the sine of the angle between them. Hence if the side lengths of 2 sides are constant, the area is maximal if they are perpendicular.