Circuits have some input capacitance. Either only parasitic from within the active elements themselves and the layout, or extra combined by a small capacitor to fix the high frequency response limit lower for stability purposes. B1 uses a JFET type with strong transconductance which is beneficial to THD and noise but it also has more internal capacitance than an average JFET. Being used in a buffer configuration there is no voltage gain so that capacitance is thankfully not multiplied.
Remember that the pot's setting combines with input capacitance to make an R-C low pass filter. So Nelson Pass knew and measured all that and he is right that the bandwidth of the circuit is much more than enough with even 100K pot.
Then, some of us experimented and we opted for no more pot value than our source equipment would feel comfortable with so to preserve more response and slew rate. 20K seemed right for a huge range of sources. We also thought that the outcome was subjectively cleaner vs 50K-100K pots that way but we are also ready to accept that it may be just a notion. On the other hand possibly it was easier for us to try decide such details on a DCB1 than on a B1 where the coupling capacitors come in the equation too.