The math function of my cheap Rigol scope reduces the 8-bit resolution to a smearing 6~7bit result, there is no fun with that.Perhaps you can measure what you want using two scope channels and the "ADD mode" function.
Just connect channel 1 probe to the amp + output and the channel 2 probe to the amp - output.
Set the sensitivity (Volts/Div) the same for both channels. Leave the scope probe ground leads open.
Set the mode to "ADD" and you should have one trace representing the output without grounding either output terminal.
I tried this with my signal generator driving a small torroidal power transformer to create a floating signal at the secondaries which were connected to the two scope channels.
Seems to work.
I use a soundcard with symmetric resistor divider for hi-resolution THD-measurements.
The transformer is my choice when measuring BTL-amps with an (oscillos-) scope
So the distortion is actually irrelevant as long as it is not so bad that it visibly affects the waveform?