what has better signal integrety over long runs of 40ft; coax or ethernet?

Ethernet cables normally handle digital signalling, while coaxial cables normally handle analogue signalling. Both will normally work fine over 40 feet providing they are presented with the connect type of signal for the type of cable. It also depends on the requirements of the interface itself. Modern Ethernet requires up to 8 conductors in the cable, while coax is normally just a single conductor inside a screened outer sheath. The original Ethernet used coax rather than the modern 8 conductor UTP leads, and was capable of handling 10 Mbps only.

It depends on the signal format, frequency and equipment involved...

With ethernet cable used for Ethernet, the specifications guarantee it to work perfectly up to a minimum of 100 metres per cable run between devices.

For coax, it depends in the type of signal; generally that's going to be used for some form of RF system, which will be attenuated according to the cable length.

Things like ADSL and some forms of cable internet use RF modems that continuously monitor the signal quality and adjust the speeds used proportionally. As the cables get longer / signal quality gets lower, they use narrower bandwidth signals that have lower losses but lower speeds.

Ethernet is simple; fixed speeds - whatever both ends of the connection are capable of from 10mbit, 100mbit & 1gbit. There is no adjustment or compensation for attenuation, it "just works" to typically well over 100m.