Just trying to figure out what you were trying to say when you said "This is certainly true at DC. But since mains is 50Hz AC then there's nowhere for the AC to go when the amp is ungrounded."
So what do you mean by "there's nowhere for the AC to go" and what exactly is the consequence of that?
OK, I agree I was using a handwaving style of discourse. So then, let's be a bit more precise. Mains supplies are referenced to earth at the substation - that's my understanding (which could be wrong as I'm not an electrical engineer). So with all double-insulated components connected together, there's the potential for the phono socket shield (assuming they're all interconnected by unbalanced cables) to be considerably above earth potential. This comes about because of the capacitance to mains provided by the transformers in the system. In this condition, there is a common-mode voltage (measured relative to earth) on all the exposed metal contacts, but no common-mode circuit and hence no way for the common-mode potential to drive an appreciable current anywhere. That was my meaning of 'nowhere for the AC to go'. Any clearer now?
So to answer your question about the consequences - they're that there's a common-mode voltage on all the cables. If there's a grounded component in the system, this common-mode voltage will be considerably reduced, but not of course totally eliminated.
How so? What exactly is the mechanism?
The noise appears across the cable for the simple reason that there's an AC current loop - an AC circuit, part of which is the cable screen. Given an EMF and a finite conductance, a current will flow. A current flowing with a finite conductance develops a potential difference.
Just for fun I measured some cables I have lying around. The first one was a freebie which came with one of my 99RMB DVD players (see my blog if you'd like to learn more). This is about 1m long and when the screen was passing 1A DC it measured almost exactly 1V end to end. So it has 1ohm screen resistance.
The next one was a 1.5m length of video cable - this dropped 80mV, hence 80mohms. The last was a 5m twin phono - stereo jack cable, measured back to back at 800mohms. So each 5m length contributes 400mohms in screen resistance.
The freebie cable is more than an order of magnitude worse than those other two cables. Quite simply - its been cost-engineered and doesn't have enough copper in the screen.
Now if this cable were used between my 99RMB DVD player and an earthed amplifier, it would be carrying the common-mode currents back to earth. I seem to recall measuring about 200uA of leakage to earth from the player - this becomes 200uV of noise (remember, the cable screen is 1ohm) and is in series with the wanted signal (2V max). The SNR is therefore only 80dB, not very impressive for a purportedly 16bit system. Actually, it would be 6dB better because we'd have the two screens in parallel, still underwhelming though.
If the amp though were double insulated, we might get a much better figure. It would probably be limited by the common mode RF performance of the amp and the cable inductance (which I don't have the means to measure).