On thinking over the sensitivity of the CS5381 to the drive and input filtering capacitor I decided to explore the capacitors further.
I was also interested to note that a number of manufacturers seem to recommend a range of different capacitor values here.
Purely because it was handy - i.e. right in front of me on the desk - I threw an extra 2nF capacitor across the differential input of the CS5381. Boom - the distortion dropped 6dB straight off.
I muttered a few choice profanities, which made me feel an awful lot better, then arbitrarily threw a 470pf NPO ceramic cap across that lot, just to be sure. Well given I was off doing such arbitrary things - why not?
On analysis, 5nF capacitance at 20kHz is about 1.5K Ohms (reactive) which is within the capability of the op amp to drive.
Which I found rather pleasing, as using the "default" 2n7,...