A pot )or fixed divider once the attenuation factor is known and tweaked) is the right way, short of:
1) modifying one of the two amps for less/more gain (hardly practical).
2) Adding a transformer or autoformer on the amp output. Driving a loudspeaker (especially if there is a passive crossover in there) normaly implies that the signal source for the speaker system (i.e. amplifier) has negligible internal impedance. Quite obviously this would not be the case, so at best you would get a lot of heat to lose, and more likely, a signifficantly changed frequency response. Therefore, in order to attenuate at the output you would actually need a transformer or autoformer.
3) With some heavy assumption regarding the actual load that has higher sensitivity, you could also modify the crossover to include the attenuation.
Now, all of the above seem far more of a cost and bother than stringing up two resistors per input channel on the amp that needs attenuation, and hence providing input attenuation. However, you did ask for the 'most audiophile' way, that would probably be something along the lines of 2) above