Something isn't adding up...
You show a notch output level, post-LNA on the first FFT at -10 dBFS. There is about 50 dB post-notch gain which would put the actual 1 kHz notch level, pre-LNA at about -60 dBFS. You also mention that the notch depth is about 50 dB. That would put the oscillator output level, and the input to the notch at -10 dBFS.
If the above is true then the reference carrier level, 0 dBc, has to be close -10 dBFS.
The level of H2 is about -168 dBFS but only -158 dBFS below the 0 dBc, (-10 dBFS) carrier.
Subtracting the stated 8 dB insertion loss at H2 yields about -150 dBc.
Slightly OT:
I read earlier about the poor performance of the OPA1642. FWIW I just recently measured the output short-circuit current of several popular duals and the OPA1642 was the lightweight of the bunch at 28 mA. The OPA2604 and MC33078 were also lame at 30 mA. The OPA1612, LME49720 and multi-vendor 5532s measured in the 36-48 mA range. The King Daddy was the NJM4560 at 60 mA, the NJM2114 a close second at 56 mA and the NJM2068 coming in third at 52 mA. I can see how the 100 mA output of the OPA1656 would be beneficial.
My 0 dBc is 2.7 Vrms 1 kHz from oscillator.
Notch is - 50 dB at 1 kHz.
LNA gain is around 30 dB (28.9 actually).
Forte input has gain 20 dB before ADC.
So we are back at about 2.7 Vrms which is around -10 dBFS of the ADC what you see on both pics. FS of the ADC is 7.75 Vrms (20 dBu).
All harmonics in REW charts are relative to the 2.7 Vrms (to our 0 dBc).
First pic is with SW compensation of the notch frequency response (imported cal file in REW). Here we see THD with all harmonics directly (with few dB of computation error).
Second one is not compensated, so we have to calculate real harmonics manually. H2 is 42 (50-8) dB down, H3 is 46 (50-4) dB down than we see on the pic relative to the dBc. Then H2 is -125-42=-167 dB, H3 is -126-46=-172 dB.
Hope, now it is more clear.
Martin