Do I need an Output Selector Switch

I have a plan for a passive-pre that consists of an input selector switch for three sources and a stepped attenuator. The sources will be a CD player and a PC for now.

I want to be able to connect it to my lm3875 amp and also my headphone amp. The amps won't be on at the same time.

Do I need a selector switch after the attenuator to switch the output between the main amp and the headphone amp? Or can I just have two sets of outputs in parallel? Is their a better way?