I do not know but that CPU load is a bug somewhere. IMO in the alsa-lib layer, maybe in the s16 scope not properly supporting all alsa stream statuses (paused, resumed, etc.)
I thought that inability of the player (mpd, vlc ...) to access the RT Scheduler caused the issue. It looks like the players need that scheduler when they are caching data from the network stream. Maybe I'm missing something here. The deep debugging is not so easy to setup.
Whether it’s a bug or not, whether it’s in alsa-lib or somewhere else, what’s the difference.
There was a problem and there is a solution.
It may be not perfect, not strict, but it works.
There was a problem and there is a solution.
It may be not perfect, not strict, but it works.
There are two solutions right now - disable caching in the player and run the player with 'sudo'. I prefer the former one. I think the solution with required privileges for the 'pi' user would be the best. Still need to search for missing privileges...
Why it happens when the 'peppyalsa' plugin is in use? Probably because it's also runnig with 'pi' user (?)
Why it happens when the 'peppyalsa' plugin is in use? Probably because it's also runnig with 'pi' user (?)
>CPU load is a bug somewhere
What I saw several times is a specific thread synchronization made via spin locks.
It’s not a pure mutex, however mutex inside is nearly the same spin lock, in Windows at least.
Switching thread context is time consumptive, so when the mutex (spin lock) is busy, the kernel doesn’t switch the waiting thread context immediately, it just spins for a short period in the hope the synchronized object will be available soon, so it spins. If a thread faces the mutex/spin-lock often we would have the CPU spikes. Don’t you remember the only point in the suspicious alsa call implementation is list-add-tails (synchronized access to the list), so it could spin and give CPU spikes
What I saw several times is a specific thread synchronization made via spin locks.
It’s not a pure mutex, however mutex inside is nearly the same spin lock, in Windows at least.
Switching thread context is time consumptive, so when the mutex (spin lock) is busy, the kernel doesn’t switch the waiting thread context immediately, it just spins for a short period in the hope the synchronized object will be available soon, so it spins. If a thread faces the mutex/spin-lock often we would have the CPU spikes. Don’t you remember the only point in the suspicious alsa call implementation is list-add-tails (synchronized access to the list), so it could spin and give CPU spikes
I was able to get the alsa-lib 1.1.8 source files and compile it by following these simple steps:
Updating ALSA on Raspbian Stretch
The debugging info was enabled already in the Makefile:
CFLAGS = -g -O2
Is there anything else required ?
Updating ALSA on Raspbian Stretch
The debugging info was enabled already in the Makefile:
CFLAGS = -g -O2
Is there anything else required ?
rpi: I would recommend not to compile a new alsa-lib version as very likely the existing alsa clients will not work with the new library (at least that's what happened to me several times). Debian provides an (relatively) easy way to recompile the existing packages with minor changes. First I would try installing the package with debugging symbols to see if the perf output is contains just the hexa offfsets.
E.g. HowToGetABacktrace - Debian Wiki
I am not sure raspbian provides libasound2-debug/dbgsym packages, but they might be installable from stock debian How do you install a dbgsym package? - Raspberry Pi Forums
Perhaps easier would be rebuilding the package with the debug symbols enabled.
E.g. HowToGetABacktrace - Debian Wiki
I am not sure raspbian provides libasound2-debug/dbgsym packages, but they might be installable from stock debian How do you install a dbgsym package? - Raspberry Pi Forums
Perhaps easier would be rebuilding the package with the debug symbols enabled.
Gentlemen, pardon me, imho you are going a wrong way.
If mpd (and probably the others) decided to use the thread priorities, they were not able to fix the problem on the alsa side. However you decided to start even from a point earlier – profiling alsa.
Imagine, even if you find a suspicious point in alsa (very likely rpi has already found it accurate to the function) , what you would do then? Changing the alsa threading model? Something similar?
Do you really have enough time and patience to accomplish that?
Is there really no chance to grant pi-user enough permissions to manipulate with the priorities?
It’s not a utopia, mpd-user/audio-group are provided with such permissions, so the method exists for sure. As a last resort mpd-user can be used for running players.
The decision is obviously up to you, it’s just my humble opinion.
Rpi: Do you still need my practical (testing) assistance? If my (testing) “mission” is over I would return my PI4 to the “usual life” back. Please don’t get me wrong I would be happy to participate in the discussion further, just would like to assemble the PI4 back.
If mpd (and probably the others) decided to use the thread priorities, they were not able to fix the problem on the alsa side. However you decided to start even from a point earlier – profiling alsa.
Imagine, even if you find a suspicious point in alsa (very likely rpi has already found it accurate to the function) , what you would do then? Changing the alsa threading model? Something similar?
Do you really have enough time and patience to accomplish that?
Is there really no chance to grant pi-user enough permissions to manipulate with the priorities?
It’s not a utopia, mpd-user/audio-group are provided with such permissions, so the method exists for sure. As a last resort mpd-user can be used for running players.
The decision is obviously up to you, it’s just my humble opinion.
Rpi: Do you still need my practical (testing) assistance? If my (testing) “mission” is over I would return my PI4 to the “usual life” back. Please don’t get me wrong I would be happy to participate in the discussion further, just would like to assemble the PI4 back.
Realtime priorities are used to minimize danger of xruns for low-latency playback (i.e. when setting small alsa buffers). That has nothing to do with fixing some (potential) bugs in (likely poorly tested) alsa meter/scope layer.
Of course any user can be allowed to set the realtime priority, a link to the procedure has been here for a while. But that is an unreliable (basically random) workaround.
Of course any user can be allowed to set the realtime priority, a link to the procedure has been here for a while. But that is an unreliable (basically random) workaround.
>But that is an unreliable (basically random) workaround.
Undoubtedly the workaround is not too reliable, but practically spread widely enough. The spikes related problem is purely practical
Undoubtedly the workaround is not too reliable, but practically spread widely enough. The spikes related problem is purely practical
But the realtime priority setup in players is used for a totally different reason, it has nothing to do with the problem you are hitting. The fact that it "helps" in your specific case is purely coincidental.
>realtime priority setup in players is used for a totally different reason
Could you please explain the "totally different reason"
Could you please explain the "totally different reason"
These are interesting articles related to realtime audio
Audio Latency demystified, part 1/4 - MINDMusicLabs.com
Audio Latency demystified, part 2/4: Real-time Linux approaches - MINDMusicLabs.com
Ross Bencina >> Real-time audio programming 101: time waits for nothing
Audio Latency demystified, part 1/4 - MINDMusicLabs.com
Audio Latency demystified, part 2/4: Real-time Linux approaches - MINDMusicLabs.com
Ross Bencina >> Real-time audio programming 101: time waits for nothing
Hi Serge, thank you for your time and help! Is it possible to find out which permissions the 'mpd' has but 'pi' doesn't?
phofman, I played with different settings/combinations in 'rtprio' for the 'pi' user. It didn't help.
I'm not sure that I'll be patient/knowledgeable enough to continue debugging 🙂 I will probably spend some time during the weekend.
Thank you all!
phofman, I played with different settings/combinations in 'rtprio' for the 'pi' user. It didn't help.
I'm not sure that I'll be patient/knowledgeable enough to continue debugging 🙂 I will probably spend some time during the weekend.
Thank you all!
The realtime setup was not definitely incorporated into mpd to fix the high-load problem in alsa meter layer. The fact that it may cause the faulty condition to not arise is pure coincidence with regard to the purpose of the realtime setup.
A tiny step forward was made
after
sudo setcap CAP_SYS_NICE,CAP_SYS_ADMIN+eip /usr/bin/mpd
( CAP_SYS_ADMIN probably redundant)
mpd shows no exceptions in mpd.log, but the spikes are still there.
Looks like starting from mpd with
/etc/systemd/user/default.target.wants/mpd.service
grants more permissions
Edit
There is a related link
real time - How to set CAP_SYS_NICE capability to a Linux user? - Stack Overflow
mpd.service has
LimitRTPRIO=50
LimitRTTIME=infinity
So far I couldn’t find anything audible for what systemd does
after
sudo setcap CAP_SYS_NICE,CAP_SYS_ADMIN+eip /usr/bin/mpd
( CAP_SYS_ADMIN probably redundant)
mpd shows no exceptions in mpd.log, but the spikes are still there.
Looks like starting from mpd with
/etc/systemd/user/default.target.wants/mpd.service
grants more permissions
Edit
There is a related link
real time - How to set CAP_SYS_NICE capability to a Linux user? - Stack Overflow
mpd.service has
LimitRTPRIO=50
LimitRTTIME=infinity
So far I couldn’t find anything audible for what systemd does
Serge, thank you for the update. What happens if you comment out lines
LimitRTPRIO=50
LimitRTTIME=infinity
when you run as 'mpd' user?
In my case 'mpd' belongs only to the 'audio' group. Is it the same for you?
$ groups mpd
mpd : audio
LimitRTPRIO=50
LimitRTTIME=infinity
when you run as 'mpd' user?
In my case 'mpd' belongs only to the 'audio' group. Is it the same for you?
$ groups mpd
mpd : audio