Yep, a few samples delay on their own are totally inaudible, however a few samples delay between woofer and tweeter might totally screw up the phase response of a otherwise properly designed crossover! And that might be totally audible as it could also introduce devere cancellations at the crossover frequency. I keep saying "might" because it all depends on the actualy amount of delay.