Dolby Atmos Track on a 5.1 channel system?

The optical cable may have some limiting effect ,An HDMI cable should give a better sound quality signal than optical ..even with a 5.1 speaker system...The quality of the AV processor inside your AV receiver is the deciding factor, Not all processors are of the same quality, and usually a higher end AV unit will have a better processor than a lower end unit, but not always, sometimes they are the same exact processor.... A better processor will not only do a better job in atmos but will also process a 5.1 output more smoothly and with better detail and dynamics. Most of that is only going to come through HDMI..though

No, your receiver will default to a 5.1 setting that it can handle.

It will. Dolby Laboratories Atmos is engineered to use an array of overhead speakers in addition to your pre-existing plane-of-ears speakers; I consider 5.1.4 minimum for adequate directional quality. Dolby recommends sky-firing speakers at the corners for height sounds with low ceilings; some sky-firers may be available from the manufacturer of your existing corner speakers.

The system will only use those audio channels you actually have.

The source doesn't "try to use" any number of speakers.

Systems that can't decode more than 5.1 channels

are not affected by signals with more channels encoded in them.

I wouldn't think so. the track you refer to is an option for those few who bother buying an Atmos system and the associated speakers.