I am completely certain that I am missing something here with regard to the bigger picture.
There is no question that FFT's have limits, especially in actual practice. Just as with many other test measurement techniques.
But, how much variation from "truth" are we talking about in audio applications? A dB in some cases? Five? A tenth?
OK, so we can all debate about that. Just as we can debate whether that difference can be heard by humans in a listening room. The latter probably shouldn't be discussed in this thread...
But, we have just seen a couple examples of where the ~22 KHz noise floor changed by ~20 dB due to reducing the effects of some sort of interfering currents. I'd think that it's plausible to consider that this level of change would be audible. Perhaps that might be described as veiled or grainy by some.
Someone please help me out here. We can have somewhat imperfect and limited measurements - often limited by sticking to the same test regimes rather than really looking at what we have in front of us - or we can dismiss it all because it isn't perfect or close to it. Which?
As for Measuring Distortion on the Cheap, my own purely subjective opinion is that there's more to be learned from looking beyond the boundaries with less professional test gear than there is by trying to scratch another tenth of a dB in resolution in our traditional measurements. That doesn't mean that Audio Precision, despite some of their business aspects we might not like, doesn't have a big place in the audio world - it does. But, the hobbyists can do a lot, too. Think of the discoveries made by amateur astronomers with modest telescopes, at least compared to the big better funded - ok, maybe formerly better funded - observatories.
There is no question that FFT's have limits, especially in actual practice. Just as with many other test measurement techniques.
But, how much variation from "truth" are we talking about in audio applications? A dB in some cases? Five? A tenth?
OK, so we can all debate about that. Just as we can debate whether that difference can be heard by humans in a listening room. The latter probably shouldn't be discussed in this thread...
But, we have just seen a couple examples of where the ~22 KHz noise floor changed by ~20 dB due to reducing the effects of some sort of interfering currents. I'd think that it's plausible to consider that this level of change would be audible. Perhaps that might be described as veiled or grainy by some.
Someone please help me out here. We can have somewhat imperfect and limited measurements - often limited by sticking to the same test regimes rather than really looking at what we have in front of us - or we can dismiss it all because it isn't perfect or close to it. Which?
As for Measuring Distortion on the Cheap, my own purely subjective opinion is that there's more to be learned from looking beyond the boundaries with less professional test gear than there is by trying to scratch another tenth of a dB in resolution in our traditional measurements. That doesn't mean that Audio Precision, despite some of their business aspects we might not like, doesn't have a big place in the audio world - it does. But, the hobbyists can do a lot, too. Think of the discoveries made by amateur astronomers with modest telescopes, at least compared to the big better funded - ok, maybe formerly better funded - observatories.
There's lots of discussion about this kind of immunity testing in professional EMI/RFI circles. Lots of gear available, too.
But, being a hobbyist, I like to build my own test accessories when I can. I learn a lot and save a bunch of money. The trade off is lack of official traceability, which is obviously important for a lot of uses. But, I can make comparative measurements ("did that change make things better or worse?") and come pretty close to what the pros get if I am careful and somewhat clever about my calibrations. In the end, to whom am I accountable to for these measurements? My wife? My dog?
In that spirit, I'll offer up this DIY LISN device, which can be used for monitoring or injecting. (I am NOT the author.)
Gary Johnson DIY LISN Device
Combined with one of these, which I do own and have good luck with, you can make measurements from 50 KHz up to beyond a GHz.
NanoVNA V2
There are no issues with FFT, other than inability of some members to understand it.
It could be 100dB or more. Suppose you take a high resolution FFT that takes 10seconds to acquire. Suppose further that there is a full scale burst of chirp noise for 0.1 seconds. In the best case for measurement let's say the chirp noise is primarily in one FFT bin. The correlation of the burst signal with the FFT bin frequency will be reduced by 100 times due to the short duration of the noise. Now, there will probably be some other FFT components produced, but with a chirp signal they will be minimal. The point is that the spectral line for the noise will be 100 times lower than the peak noise level simply because of its short duration, so some people will say it was probably inaudible or negligible.
Of course, this is not the usual case where FFT is not that useful. Its just that you asked about how far off it could be, so I gave an example that produces a large-ish difference.
That said, if you are looking for time domain transients with FFTs, then better to use short-time FFTs. However, frequency resolution will be limited.
A more difficult case would when looking for transients in spectral line noise skirts. You need high resolution in the frequency domain and in the time domain, both at the same time.
To summarize, the issues with FFTs result from three main causes: (1) FFTs are an integral transform so there is an averaging effect, (2) phase information is discarded in spectral graph views (as pointed out by Purifi's Lars Risbo, a PhD. with dozens of patents), and (3) the FFT accuracy will be no better than the accuracy of the ADC, which might be less accurate or no more accurate than the DUT.
Here is the example from Risbo: https://purifi-audio.com/blog/tech-notes-1/doppler-distortion-vs-imd-7 ...The two audio files have the exact same spectral FFT view, but do they sound alike on headphones?
Last edited:
Hi Mark,
I am careful to conduct tests following industry accepted practices. Even my dummy loads are the specific audio industry standard. When I perform testing not under any standard form, they are documented and involve the people directly involved. If anyone wants to know how a test is conducted, easily found by learning standard industry "best practices".
Often, it has been clamed people who do testing don't actually listen and assess their work. This is a false assumption. I merely pointed out that not only have I invested in test equipment, but also heavily in audio reproduction systems. Good techs do both, measure and listen. Also, what goes on between your ears will easily overrode what you experience in listening tests. Not a recent observation but one borne over decades of observation.
When I was a baby technician, we couldn't measure very well, nor did we really understand what to measure for. So comments to that effect were accurate for the time. However the industry understanding of what to measure for, and the equipment evolved to our current state far advanced from the 1970's or earlier. Some folks hold onto those early ideas. The only reason why me and everyone else evolved was by recognizing the short comings and asking why? Then chasing down the issues. As the equipment becomes better, it leads our understanding and we can test ideas without guessing. I study instrumentation, and I have some HP 3580A audio spectrum analyzers. Amazing instruments for the time. Equipment progresses and we upgrade what we have on the bench. In addition, some of us bought component analyzers and actually studied the parts as they improved. Like a mechanic, a good tech must understand the parts they are using. One, an HP 4192A is pretty handy and is used regularly, as is an HP 4263A. It's all in search of knowledge and to confirm ideas on circuit performance.
Now, when I see someone derailing a discussion because they don't have technical proof using an argument based in psychology, that is too far. That approach must end completely. Want to discuss psychology? That is a separate thread. So is the argument that we don't know everything. We don't, but what is known is solid and well documented. Proved by several people over time. Some people believe the world is flat. Doesn't make it so, and it doesn't make them right. Yet they will argue given a chance.
I am careful to conduct tests following industry accepted practices. Even my dummy loads are the specific audio industry standard. When I perform testing not under any standard form, they are documented and involve the people directly involved. If anyone wants to know how a test is conducted, easily found by learning standard industry "best practices".
Often, it has been clamed people who do testing don't actually listen and assess their work. This is a false assumption. I merely pointed out that not only have I invested in test equipment, but also heavily in audio reproduction systems. Good techs do both, measure and listen. Also, what goes on between your ears will easily overrode what you experience in listening tests. Not a recent observation but one borne over decades of observation.
When I was a baby technician, we couldn't measure very well, nor did we really understand what to measure for. So comments to that effect were accurate for the time. However the industry understanding of what to measure for, and the equipment evolved to our current state far advanced from the 1970's or earlier. Some folks hold onto those early ideas. The only reason why me and everyone else evolved was by recognizing the short comings and asking why? Then chasing down the issues. As the equipment becomes better, it leads our understanding and we can test ideas without guessing. I study instrumentation, and I have some HP 3580A audio spectrum analyzers. Amazing instruments for the time. Equipment progresses and we upgrade what we have on the bench. In addition, some of us bought component analyzers and actually studied the parts as they improved. Like a mechanic, a good tech must understand the parts they are using. One, an HP 4192A is pretty handy and is used regularly, as is an HP 4263A. It's all in search of knowledge and to confirm ideas on circuit performance.
Now, when I see someone derailing a discussion because they don't have technical proof using an argument based in psychology, that is too far. That approach must end completely. Want to discuss psychology? That is a separate thread. So is the argument that we don't know everything. We don't, but what is known is solid and well documented. Proved by several people over time. Some people believe the world is flat. Doesn't make it so, and it doesn't make them right. Yet they will argue given a chance.
Hi Mark,
Do you even understand what an FFT is all about? A chirp is not a valid test signal for FFT. It is used in radar for example, different application.
This is the kind of thing I am talking about. Bringing in things that are not valid for these tests does not further any discussion. Pulse testing shows other things that are monitored by other instruments. A classic chirp varies level and frequency, and a pulse contains all frequencies by definition. So what is an FFT going to show you?
Do you even understand what an FFT is all about? A chirp is not a valid test signal for FFT. It is used in radar for example, different application.
This is the kind of thing I am talking about. Bringing in things that are not valid for these tests does not further any discussion. Pulse testing shows other things that are monitored by other instruments. A classic chirp varies level and frequency, and a pulse contains all frequencies by definition. So what is an FFT going to show you?
Hi Chris,
I know that the only valid test signal for a standard audio FFT is a PSS signal (which means it can only contain steady state sine waves). The problem is people use such FFTs to measure systems which are producing artifacts other than of the assumed PSS type. IMHO and in the opinion of many other people it is a serious problem that users misinterpret FFT results because the users are expecting only PSS signals to exist in the FFT results.
BTW, I also agree with @1audio when he points out a measurement problem is that that music is not PSS.
Mark
EDIT: I would also just add that chirp artifacts can be produced by some dacs under the conditions described by @MarcelvdG
I know that the only valid test signal for a standard audio FFT is a PSS signal (which means it can only contain steady state sine waves). The problem is people use such FFTs to measure systems which are producing artifacts other than of the assumed PSS type. IMHO and in the opinion of many other people it is a serious problem that users misinterpret FFT results because the users are expecting only PSS signals to exist in the FFT results.
BTW, I also agree with @1audio when he points out a measurement problem is that that music is not PSS.
Mark
EDIT: I would also just add that chirp artifacts can be produced by some dacs under the conditions described by @MarcelvdG
Last edited:
Well, non-averaged tests show all kinds of noise, as do averaged tests where noise will bump out skirts and the noise floor. An FFT does have some minimal averaging (sample length) and people have to understand that too.
But a steady state sine wave is misunderstood by many. A sine represents all levels up to the peak, so it exercises the system. Then, running a dual sine (I use 19 KHz and 20 KHz) is a rather more severe test as feedback is at a lower level at higher frequencies. This is more likely to generate intermodular distortion and that shows up like a sore thumb. Music is no more challenging for a system than a sine or two. Systems don't have a memory as we normally think of it. Thermal issues could be, but any system has no way to predict what is coming in the next instant. Tone bursts will test for thermal memory and voltage collapse easily.
But you can't impugn the standard test equipment and methods just because some don't understand the equipment or what they are doing. A car isn't evil, never mind how many people mis-use them or don't understand them and use them anyway.
It really boils down to understanding your tools and their limitations, and also how to use those tools properly. Interpreting an FFT is also a skill, knowing what the small details are telling you. Like any experiment (which is what you are doing), you have to design the experiment to only allow variables you are looking at effect the results. You have to control everything else - or your results are not valid. This is where skill in setting things up come in.
But a steady state sine wave is misunderstood by many. A sine represents all levels up to the peak, so it exercises the system. Then, running a dual sine (I use 19 KHz and 20 KHz) is a rather more severe test as feedback is at a lower level at higher frequencies. This is more likely to generate intermodular distortion and that shows up like a sore thumb. Music is no more challenging for a system than a sine or two. Systems don't have a memory as we normally think of it. Thermal issues could be, but any system has no way to predict what is coming in the next instant. Tone bursts will test for thermal memory and voltage collapse easily.
But you can't impugn the standard test equipment and methods just because some don't understand the equipment or what they are doing. A car isn't evil, never mind how many people mis-use them or don't understand them and use them anyway.
It really boils down to understanding your tools and their limitations, and also how to use those tools properly. Interpreting an FFT is also a skill, knowing what the small details are telling you. Like any experiment (which is what you are doing), you have to design the experiment to only allow variables you are looking at effect the results. You have to control everything else - or your results are not valid. This is where skill in setting things up come in.
This thread is a useful resource for those who want to test audio using methods available via REW / AP etc (however flawed). If it's not too much bother, maybe the posts discussing the validity of FFT measurement could be moved to a new thread? There are a gazillion audio forums that talk about audio in terms of mP (magic pixies) instead of dB. If this is the 'equipment and tools' forum, maybe it ought to keep things objective?
That's quite possibly the best summary of FFT I've read to date. Thank you.
But this also means that a signal can't hide between FFT bins as was suggested earlier. If it's in the original time-domain signal, it will be represented in one or more FFT bins.
If the sampled signal is not periodic within the FFT window, the windowing function will "force it" to be by fading in/out at the beginning/end of the FFT window. But applying a window function changes the original samples, so you have issues with side lobes and whatnot. That's a topic for its own separate severed thread (see what I did there? 🙂). One can play with "no window" vs other windows by choosing a rectangular window (= no window) or any of the other window functions in the analyzer software.
This also further hammers home why you would want the DAC and the ADC to be on the same clock, which is another topic we discussed earlier. If they're on the same clock, you can guarantee that the test signal will be periodic within the FFT window. If the ADC and DAC are on different clocks, even if they're the same frequency, it can be guaranteed that the test signal will not be periodic within the FFT window, because two oscillators will never have the exact same frequency.
Tom
Of course it cannot hide. The suggestion resulted from incompetence of the poster. As did/do many other opinions on FFT. I second to the idea of moving opinions on FFT inadequacy away.
I've requested that the FFT discussion (and possibly the multi-tone discussion as well) be moved. Suggested branch points: Post 201 and 353.
Tom
Tom
This one to be moved as well. But I can't help to show that the idea of disappearing spectral content is a nonsense, demonstrated on 256 points FFT. It is just as @JohnPM has mentioned.
Understanding the limitations of test equipment is exactly about being objective.
Sure. But the "limitations" you've described in FFT so far have been rooted in your lack of understanding of how FFTs work. Those "limitations" aren't actually there.
Similarly, the "limitations" you have clamoured about regarding distortion are subjective (can/cannot hear, does/does not sound good) rather than objective.
Tom
Similarly, the "limitations" you have clamoured about regarding distortion are subjective (can/cannot hear, does/does not sound good) rather than objective.
Tom
And what about 1audio quote above? Big difference even if measured results are similar. Not objective?
