Some tools to measure parts of the jitter...I will see how much I can apply myself, but it shows that the phenomen is real and much smarter people have even developed tools to measure software and hardware related jitter under Linux:
https://git.kernel.org/pub/scm/linu...t/tree/Documentation/trace/osnoise-tracer.rst
https://github.com/rigtorp/hiccups
https://rigtorp.se/low-latency-guide/
https://git.kernel.org/pub/scm/linu...t/tree/Documentation/trace/osnoise-tracer.rst
https://github.com/rigtorp/hiccups
https://rigtorp.se/low-latency-guide/
(From H. Akkan, M. Lang, L.M. Liebrock, Stepping towards noiseless Linux environment)
...so tickless would be quiet interesting...with the minimum in jitter and noise....which is controlled by the parameter:
https://www.suse.com/c/cpu-isolation-nohz_full-part-3/
I tried that on my Odroid xu4, but did not do anything in interrupts differently, I still have the same mcktimer interrupts as before on those core, so need to figure out why the parameter did not work...ARM-specific ?
...so tickless would be quiet interesting...with the minimum in jitter and noise....which is controlled by the parameter:
https://www.suse.com/c/cpu-isolation-nohz_full-part-3/
I tried that on my Odroid xu4, but did not do anything in interrupts differently, I still have the same mcktimer interrupts as before on those core, so need to figure out why the parameter did not work...ARM-specific ?
OK, I found
ARM can do ticklesss, most distros have not been compiled with this option though
So I yet have to find a distro where it is enabled (search was so far not successful, so if anyone has a hint...). Otherwise I have to compile my own...but tickless still is an important topic to fight jitter if you do research.
Type of USB-implementation can generate a lot of noise/interrupts
On dietpi: I managed to isolate the CPU with isolcpus in the Kernel command and observe now the interrupts on each board and go through the old posts of Tuxx, the inventor of Archphile. It looks like that each board has both, different USB-realizations as well as I2S, read: Some have outsourced certain USB-processing to the CPUs and produce 5000-10000 interrupts per second, no matter if I play a 44k or 192k pcm file...there seems to be solutions where the (more epensive) USB-chip does this by itself and others where the CPU is used instead...polluting it with interrupts. I will now analyze the 12 boards I have here on that and look up what the interrupt behavior on each board is. Anyway I got now USB, ethernet, MPD/audio and Linux-System interrupts all on different isolated CPUs.
I2S-implemetation can vary as well a lot.
Some have their I2S hardcoded to be 48khz as its is meant to go over hdmi out etc.etc. A topic for later.
WAVE (0.7% Cpu Utilzation) vs. FLAC (5% CPU Utilization)
When I use Flac-Files to be played without any other processing, I get approx 5% CPU-utilzation on the audio core. I you go and anaylze this further on sub-process/interrupt level you find 5 subprocesses of audio/MPD with MPD and Flac the big ones, each 2-3% utilization. Now, the interesting part: When I play a Wave-file MPD drops to 0.7%. That means flac and MPD fight for CPU-Resources on an isolated Core. Need to do listening tests. But he concept of an isolated core is to isolate application processes as well, ideally one app per core...so i might convert my Music to wave.
To Real-time or not to Real-time
Real-time systems are designed for a specitfic purpose...for realtime situations like a Musician hits a key and a tone needs to be there in realtime. Or controlling the motor electronic of a car. This requires fast CPUs, 1000Hz interrupt timers, a kernel with a different scheduler which is designed to be in time etc.:
https://www.researchgate.net/public...Timing_Jitter_and_its_Impact_on_Motor_Control
So, Real-TIme are designed for low latency meaning design target is faster, many more Interrupts, higher clocks, lower buffers....and lower jitter.
Lower latency is not what we need to playback an audio-file though. Lower latency systems mean lower buffers, higher switching times more interrupts, more noise. Therefore tickless is even more important for realtime-systems.
We want lowest jitter AND lowest noise. MAny people reported that actually higher latencies on slower machines sound best...which can makes sense as interrupts are reduced etc. I found lowest latency setting to sound dry, like a strong Global Feedback amp.
So, maybe I need to compile a kernel which brings all the good stuff from Real-Time-Kernels like in time scheduling policies, but design the rest like scheduling timer to 100Hz as we want lowest noise as well. ..and compile another one without Real-time or with 1000hz.
ARM can do ticklesss, most distros have not been compiled with this option though
So I yet have to find a distro where it is enabled (search was so far not successful, so if anyone has a hint...). Otherwise I have to compile my own...but tickless still is an important topic to fight jitter if you do research.
Type of USB-implementation can generate a lot of noise/interrupts
On dietpi: I managed to isolate the CPU with isolcpus in the Kernel command and observe now the interrupts on each board and go through the old posts of Tuxx, the inventor of Archphile. It looks like that each board has both, different USB-realizations as well as I2S, read: Some have outsourced certain USB-processing to the CPUs and produce 5000-10000 interrupts per second, no matter if I play a 44k or 192k pcm file...there seems to be solutions where the (more epensive) USB-chip does this by itself and others where the CPU is used instead...polluting it with interrupts. I will now analyze the 12 boards I have here on that and look up what the interrupt behavior on each board is. Anyway I got now USB, ethernet, MPD/audio and Linux-System interrupts all on different isolated CPUs.
I2S-implemetation can vary as well a lot.
Some have their I2S hardcoded to be 48khz as its is meant to go over hdmi out etc.etc. A topic for later.
WAVE (0.7% Cpu Utilzation) vs. FLAC (5% CPU Utilization)
When I use Flac-Files to be played without any other processing, I get approx 5% CPU-utilzation on the audio core. I you go and anaylze this further on sub-process/interrupt level you find 5 subprocesses of audio/MPD with MPD and Flac the big ones, each 2-3% utilization. Now, the interesting part: When I play a Wave-file MPD drops to 0.7%. That means flac and MPD fight for CPU-Resources on an isolated Core. Need to do listening tests. But he concept of an isolated core is to isolate application processes as well, ideally one app per core...so i might convert my Music to wave.
To Real-time or not to Real-time
Real-time systems are designed for a specitfic purpose...for realtime situations like a Musician hits a key and a tone needs to be there in realtime. Or controlling the motor electronic of a car. This requires fast CPUs, 1000Hz interrupt timers, a kernel with a different scheduler which is designed to be in time etc.:
https://www.researchgate.net/public...Timing_Jitter_and_its_Impact_on_Motor_Control
So, Real-TIme are designed for low latency meaning design target is faster, many more Interrupts, higher clocks, lower buffers....and lower jitter.
Lower latency is not what we need to playback an audio-file though. Lower latency systems mean lower buffers, higher switching times more interrupts, more noise. Therefore tickless is even more important for realtime-systems.
We want lowest jitter AND lowest noise. MAny people reported that actually higher latencies on slower machines sound best...which can makes sense as interrupts are reduced etc. I found lowest latency setting to sound dry, like a strong Global Feedback amp.
So, maybe I need to compile a kernel which brings all the good stuff from Real-Time-Kernels like in time scheduling policies, but design the rest like scheduling timer to 100Hz as we want lowest noise as well. ..and compile another one without Real-time or with 1000hz.
Hi there Blitz,
I was not aware of this tickless thing.. It's very interesting, and might explain (at least in part) why I (and many, many more audiophiles) have had so much luck with ArchLinux - based distros.
If I am seeing correctly, it is enabled by default in Arch Linux ARM: https://github.com/archlinuxarm/PKGBUILDs/blob/master/core/linux-armv7/config
(look for CONFIG_NO_HZ_IDLE , referenced from https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt )
I was not aware of this tickless thing.. It's very interesting, and might explain (at least in part) why I (and many, many more audiophiles) have had so much luck with ArchLinux - based distros.
If I am seeing correctly, it is enabled by default in Arch Linux ARM: https://github.com/archlinuxarm/PKGBUILDs/blob/master/core/linux-armv7/config
(look for CONFIG_NO_HZ_IDLE , referenced from https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt )
Hi Dimitris,
Unfortunately that is only the setting if the CPU is idle (every distro has this enabled for laptop-powersafe mode as interrupts cost energy). Not sure when we have an idle state...if that is also the case when we have like 1% Cpu utilization or idle means idle.
The setting we want is: # CONFIG_NO_HZ_FULL and that is as well not set in archphile: # CONFIG_NO_HZ_FULL is not set
"CONFIG_NO_HZ_FULL is set" you dont find easily...maybe some RT-Kernels distros have it as RT means normally much higher amount of noise and interrupts, so you have to do something against that and reserve ideally one core per interrupt and process... which I already included now in my version through isolcpus plus setting affinity for interrupts and process MPD/audio.
...There are some more settings on RCU and memory management and for Intel Cpu as well some more NUMA-isolation tricks. which are needed for true tickless mode...but I ensure you: Now that I know about it, I will figure it out. Only a question of time.
Unfortunately that is only the setting if the CPU is idle (every distro has this enabled for laptop-powersafe mode as interrupts cost energy). Not sure when we have an idle state...if that is also the case when we have like 1% Cpu utilization or idle means idle.
The setting we want is: # CONFIG_NO_HZ_FULL and that is as well not set in archphile: # CONFIG_NO_HZ_FULL is not set
"CONFIG_NO_HZ_FULL is set" you dont find easily...maybe some RT-Kernels distros have it as RT means normally much higher amount of noise and interrupts, so you have to do something against that and reserve ideally one core per interrupt and process... which I already included now in my version through isolcpus plus setting affinity for interrupts and process MPD/audio.
...There are some more settings on RCU and memory management and for Intel Cpu as well some more NUMA-isolation tricks. which are needed for true tickless mode...but I ensure you: Now that I know about it, I will figure it out. Only a question of time.
Last edited:
Wow...they included into the new kernel even a new standard OSnoise measurement and benchmark system which anyone can use to see what does what...this is extemely nice...meed to learn it once i compiled my own kernel:
https://bristot.me/files/research/papers/osnoise/osnoise.pdf
https://bristot.me/files/research/papers/osnoise/osnoise.pdf
Maybe I should change to the other big Linux Audio-thread or start a new one....as not a lot of reaction/discussion seem to go on here as the original Archphile project seems to be dead.
TO close the Loop, I did some listening to various small experiments and here are my completely first, subjective impressions:
PSU:
Without a low noise linear PSU, dont even try to start to try differences. this is a prerequisite...I found that even the DC-DC-converter in Powerbanks sound to bad, so I am now at an LDO based PSU for the Minimim z83 I am using at the moment. Unfortunately these Mini-Board want up to 2A current while booting and later go with 0.5A or less...so Salas Shunt is not an option.
ISOLCPUS:
I isolatied 3 cores and let the Linux system run on the fourth. So, we got 1 for ethernet, 1 for USB, 1 for MPD/Audio:
This is a nice one. Backround is more black, more silent Microdetials and dynamics improve.
FLAC vs. Wave:
Without ISOLCPUS: No difference. With ISOLCPUS enabled: Another small steps towards more precise detail, better separation and focus of 3D
Network vs. local emmc storage of files (WAVE):
To my surprise quiet audible...same direction as above, network seem to add diffusion. I will switch some cables now and go to gigabit and see what this does.
Real-Time-Patch (with IsolCPUS):
Stiffer sounding, brighter sounding, a bit more technical as well but again as well more precise on microdetails...if we can get the technical sounding element away and get warmth into it with the accuracy introduced with this patch: A clear winner...otherwse a matter of taste. I would not say automatically that Rea-Time is the only way.
I just finished compilng a new Linux-Kernel for the Z83 (Atom 8350 CPU) based on Dietbi/Debian Bullseye v5.19 with the following features:
So, I will now play with many different parameter at Kernel-Level/Interrupt-Level and see what has a sensitivity on sound. I got two Z83 (70 Euro each...an x86 which is cheap than a Rpi), so I will soundwise always A/B before signing off something as a true improvement....
Parameter to look into:
TO close the Loop, I did some listening to various small experiments and here are my completely first, subjective impressions:
PSU:
Without a low noise linear PSU, dont even try to start to try differences. this is a prerequisite...I found that even the DC-DC-converter in Powerbanks sound to bad, so I am now at an LDO based PSU for the Minimim z83 I am using at the moment. Unfortunately these Mini-Board want up to 2A current while booting and later go with 0.5A or less...so Salas Shunt is not an option.
ISOLCPUS:
I isolatied 3 cores and let the Linux system run on the fourth. So, we got 1 for ethernet, 1 for USB, 1 for MPD/Audio:
This is a nice one. Backround is more black, more silent Microdetials and dynamics improve.
FLAC vs. Wave:
Without ISOLCPUS: No difference. With ISOLCPUS enabled: Another small steps towards more precise detail, better separation and focus of 3D
Network vs. local emmc storage of files (WAVE):
To my surprise quiet audible...same direction as above, network seem to add diffusion. I will switch some cables now and go to gigabit and see what this does.
Real-Time-Patch (with IsolCPUS):
Stiffer sounding, brighter sounding, a bit more technical as well but again as well more precise on microdetails...if we can get the technical sounding element away and get warmth into it with the accuracy introduced with this patch: A clear winner...otherwse a matter of taste. I would not say automatically that Rea-Time is the only way.
I just finished compilng a new Linux-Kernel for the Z83 (Atom 8350 CPU) based on Dietbi/Debian Bullseye v5.19 with the following features:
- Tickless mode enabled, SO NOHZ_FULL is set in the Kernel
- OSNOISE Tracer option enabled, so I can use now https://www.kernel.org/doc/html/latest/trace/osnoise-tracer.html to do measurement on the OSNOISE when I do the configuration changes going forward
- Atom CPU-Code enabled
So, I will now play with many different parameter at Kernel-Level/Interrupt-Level and see what has a sensitivity on sound. I got two Z83 (70 Euro each...an x86 which is cheap than a Rpi), so I will soundwise always A/B before signing off something as a true improvement....
Parameter to look into:
- Timer frequency: 250 standard vs. 100 vs 1000: Noise vs. Latency vs jitter
- Make tickless work and verify with OSnoise
- Look into the different preemptive Models from no preemptive (no disruption of MPD, server setup) to Realtime: OSNoise ? Definition of sound ?
- Look into Scheduler differences and priorities
- ...
Thanks...I will open a new one once I have my first listening impressions on the new mode of true tickless...and hopefully have something positive to report from a sound quality pov.
I am working to make the new OS-Measurement tools actually work on my system like RTLA which can measure and benchmark OSNoise snd Latency...brand new...availlable since 5.15,,,unfortunately the documentation is less than optimal to non existing how to install and make everything work...but the first OSnoise measurmeents are now active since this morning...the latency measuremnt need an additional tracer, so I ned to recompile the Kernel again, which takes hours.
I am working to make the new OS-Measurement tools actually work on my system like RTLA which can measure and benchmark OSNoise snd Latency...brand new...availlable since 5.15,,,unfortunately the documentation is less than optimal to non existing how to install and make everything work...but the first OSnoise measurmeents are now active since this morning...the latency measuremnt need an additional tracer, so I ned to recompile the Kernel again, which takes hours.
Ok...following your advise, opened a new thread:
https://www.diyaudio.com/community/threads/path-to-noiseless-linux-streamer.391202/
https://www.diyaudio.com/community/threads/path-to-noiseless-linux-streamer.391202/
For those Archphile 1.19 users.....
I just recently did a full system upgrade to fix an ALSA bug I was having a problem with (alsa-lib and alsa-utils 1.2.2 -> going to 1.2.8).
All I had to do was recompile the latest MPD (I had customised to include HTTP streaming and SID decoding) and all seems to be running reliably.
I had a few other custom utilities to recompile, but that all went cleanly.
I didn't recompile the other custom recipe packages - such as ffmpeg (but not using I believe in my MPD config), but so far so good.
I just recently did a full system upgrade to fix an ALSA bug I was having a problem with (alsa-lib and alsa-utils 1.2.2 -> going to 1.2.8).
All I had to do was recompile the latest MPD (I had customised to include HTTP streaming and SID decoding) and all seems to be running reliably.
I had a few other custom utilities to recompile, but that all went cleanly.
I didn't recompile the other custom recipe packages - such as ffmpeg (but not using I believe in my MPD config), but so far so good.
You can downloaded the last image that was built - v1.19 codenamed corona here, plus the user manual. Write to your SD card with Rufus or win32 disk imager
https://drive.google.com/folderview?id=1iVzeDYSTxP-VaCtYm1PKd6KDfDdsle4E
https://drive.google.com/folderview?id=1iVzeDYSTxP-VaCtYm1PKd6KDfDdsle4E
Someone has also kindly uploaded the archphile build scripts and source PKGBUILD files here:
https://github.com/ahtrahddis/archphile_archive
https://github.com/ahtrahddis/archphile_archive
Thanks a lot Dave. And DimDim.
I am new to Linux and know nothing of Raspberry. This img will work with Udoo Quad Linux side? Or is it meant for its Raspberry part? I will write a SD card and try anyway. Recently I wrote an Arch Linux one following these instuctions here: https://archlinuxarm.org/platforms/armv7/freescale/udoo. Creation went well but unfortunately mouse and keyboard of my Udoo weren't working and therefore I could not login. Any idea why this happens? Thanks again. Best
I am new to Linux and know nothing of Raspberry. This img will work with Udoo Quad Linux side? Or is it meant for its Raspberry part? I will write a SD card and try anyway. Recently I wrote an Arch Linux one following these instuctions here: https://archlinuxarm.org/platforms/armv7/freescale/udoo. Creation went well but unfortunately mouse and keyboard of my Udoo weren't working and therefore I could not login. Any idea why this happens? Thanks again. Best