I'm saying that what you want to do, is something some of us also want to do. Use I2S for component interconnect, with components being things like USB or BT interface, DSP with multiple outputs, I2S input DACs or power amplifier as DAC. Keeping the digitally sourced music in the digital realm certainly going in and out of the DSP part...
...To actualize something like DSP speaker crossover, while keeping everything entirely in the Digital realm, it's the easiest and least expensive way I know of to get that. Pretty much plug 'n play at the finished board level - you have to put it in a case and give it DC power.
Plugging my own thread here, but I think it might be of some interest. I2S is very bad to use for long distances due to the critical signal timing. It was mainly developed for on-PCB connection (a few inches). SPDIF is a bit more robust if you're hooking up a system over a few feet... assuming you use the right TX/RX arrangement.
I2S-Hat: A Raspberry Pi Hat for SPDIF <-> I2S Communication and DSP
Keeps everything in the digital realm, and provides more processing power for DSP than anyone knows what to do with at this time (with an RPi4). The ADSP/ADAU devices don't come close to the DSP capabilities of a dedicated RPi4. It's also relatively cheap to make if you can solder SMT and don't go extreme with the BOM.