So, what if two planets are trine by degrees, but square by sign?

An out of sign aspect would have a smaller orb than the "normal" aspects in signs you'd expect them to be in. So unless it's a pretty tight trine, you wouldn't actually consider those objects to be in aspect with each other.

It's still a trine but it's called an out of sign aspect

Good question. I've never fully understood this myself. Mathematically, energetically, you've got a trine. Perhaps the out-of-sign aspect can sometimes give you extra ability; and sometimes weaken the trine? I don't know.