ok, was paying close attention, and maybe i need to observe for a longer period, but i believe the message that i get each time there is channel switch is this "prepare playback after buffer underrun" warning:
I have a chuncksize of 1024, queuelimit 4, adjust period 10 and target level 1024. I know the tricks to minimize buffer underruns, but still wanted to know if you think this could be the cause. i will proceed now to change the settings to minimize buffer underruns and see if that minimizes channel jumps too.
I believe i did not mention this before, but the level bars (vu meter) of camilladsp does not change when the channels switch. The level bars are always consistent with what it is supposed to be, and output matches input.
I have a chuncksize of 1024, queuelimit 4, adjust period 10 and target level 1024. I know the tricks to minimize buffer underruns, but still wanted to know if you think this could be the cause. i will proceed now to change the settings to minimize buffer underruns and see if that minimizes channel jumps too.
I believe i did not mention this before, but the level bars (vu meter) of camilladsp does not change when the channels switch. The level bars are always consistent with what it is supposed to be, and output matches input.
Last edited:
ok, increased the chuncksize and the target level to avoid buffer underruns, have been listening for 50 minutes now and no channel switch so far!!!
How come that buffer underruns have this effect? are channels reassigned randomly when it happens? Do you think there is a way to avoid it?
How come that buffer underruns have this effect? are channels reassigned randomly when it happens? Do you think there is a way to avoid it?
Last edited:
When a buffer underrun occurs, camilladsp calls snd_pcm_prepare() to recover, https://www.alsa-project.org/alsa-doc/alsa-lib/group___p_c_m.html#ga788d05de75f2d536f8443cb0306754d0
It seems like this device doesn't like that. After this, some pointer isn't updated correctly. Since samples are stored interleaved, messing up the start address of a buffer could give a mixed up channel order (as long as the error is a number of full samples, if it's not then the sound would be garbage).
This looks like a driver bug to me, but it may be possible that it can be avoided by recovering in a different way. It would be worth to try snd_pcm_recover(): https://www.alsa-project.org/alsa-doc/alsa-lib/group___p_c_m.html#ga2157aaeb6fc14da3f040d76591f9d3b1
@phofman what do you think?
It seems like this device doesn't like that. After this, some pointer isn't updated correctly. Since samples are stored interleaved, messing up the start address of a buffer could give a mixed up channel order (as long as the error is a number of full samples, if it's not then the sound would be garbage).
This looks like a driver bug to me, but it may be possible that it can be avoided by recovering in a different way. It would be worth to try snd_pcm_recover(): https://www.alsa-project.org/alsa-doc/alsa-lib/group___p_c_m.html#ga2157aaeb6fc14da3f040d76591f9d3b1
@phofman what do you think?
Maybe testing how aplay deals with xruns in this particular case would tell more.
@MRamone: Please check the device buffer and period size when playing the multichannel file through CDSP (and getting the buffer issues) - /proc/asound/card/pcmXp/subxxx/hw_params and use aplay with the same params (--buffer-size, --period-size + add -v param to list the values used to check).
The goal is to hit the same xruns CDSP hits (your previous aplay command used default buffer/period values which are much larger than requested for CDSP) and to check if the same channel swap occurs. If not, we can learn from the way aplay handles xruns. It uses snd_pcm_prepare but maybe there would be some differences upon closer review...
@MRamone: Please check the device buffer and period size when playing the multichannel file through CDSP (and getting the buffer issues) - /proc/asound/card/pcmXp/subxxx/hw_params and use aplay with the same params (--buffer-size, --period-size + add -v param to list the values used to check).
The goal is to hit the same xruns CDSP hits (your previous aplay command used default buffer/period values which are much larger than requested for CDSP) and to check if the same channel swap occurs. If not, we can learn from the way aplay handles xruns. It uses snd_pcm_prepare but maybe there would be some differences upon closer review...
will do and report back, thanks both for the suggestions.
And besides all that, because i am pretty confident i can minimize underruns to almost zero, but nevertheless, if it is a driver bug, do you know any other driver that allows me to use one of this hdmi cards? my objective is to tap the i2s and have a hdmi to 4xstereo spdif outputs. The hardware part is already sorted out and working, but i wasn't expecting this little driver issue... (must work with 24 bits as that is the way i set up the hardware)
And besides all that, because i am pretty confident i can minimize underruns to almost zero, but nevertheless, if it is a driver bug, do you know any other driver that allows me to use one of this hdmi cards? my objective is to tap the i2s and have a hdmi to 4xstereo spdif outputs. The hardware part is already sorted out and working, but i wasn't expecting this little driver issue... (must work with 24 bits as that is the way i set up the hardware)
The driver is part of the kernel, so you could try upgrading to a newer kernel.
snd_pcm_recover() doesn't seem very promising. It basically just calls prepare:
https://github.com/alsa-project/als...f5661cccb5b8ba11f7a8a6e53/src/pcm/pcm.c#L8662
Maybe could be worth to try a call to snd_pcm_reset().
snd_pcm_recover() doesn't seem very promising. It basically just calls prepare:
https://github.com/alsa-project/als...f5661cccb5b8ba11f7a8a6e53/src/pcm/pcm.c#L8662
Maybe could be worth to try a call to snd_pcm_reset().
returning to the original parameters that resulted in numerours underruns i get:
marcosch@raspcamilla:~$ cat /proc/asound/card0/pcm0p/sub0/hw_params
access: MMAP_INTERLEAVED
format: IEC958_SUBFRAME_LE
subformat: STD
channels: 8
rate: 96000 (96000/1)
period_size: 256
buffer_size: 2048
if i understand it well, then i need to do:
aplay -c 8 -t wav -r 48000 -f S24_LE --period-size=256 --buffer-size=2048 -v -D iec958:vc4hdmi0 ~/8_Channel_ID.wav
can't do it now, the family is watching tv, but will try tomorrow.
marcosch@raspcamilla:~$ cat /proc/asound/card0/pcm0p/sub0/hw_params
access: MMAP_INTERLEAVED
format: IEC958_SUBFRAME_LE
subformat: STD
channels: 8
rate: 96000 (96000/1)
period_size: 256
buffer_size: 2048
if i understand it well, then i need to do:
aplay -c 8 -t wav -r 48000 -f S24_LE --period-size=256 --buffer-size=2048 -v -D iec958:vc4hdmi0 ~/8_Channel_ID.wav
can't do it now, the family is watching tv, but will try tomorrow.
The problem is the HDMI peripheral is part of the broadcom VC4 closed-source GPU, and linux kernel communicates with this device via some messages. The VC4 provides important parts - HDMI, audio PWM, PLLs, booting, etc. The VC4 is programmed by closed-source firmware which only RPi (and Broadcom, of course) people have access to. RPi is far from an open source device, in fact it's an ugly mix of closed source HW and linux running on the (basically secondary) ARM cores. Other ARM SBCs with standard ARM graphics are much more open than RPi. RPi wins in the area of software support but the HW design sucks big time.
@phofman
I tested your suggestion:
The first time, i had several buffer underruns, this is what was reported:
marcosch@raspcamilla:~$ aplay -c 8 -t wav -r 48000 -f S24_LE --period-size=256 --buffer-size=2048 -v -D iec958:vc4hdmi0 ~/8_Channel_ID_long.wav
Warning: format is changed to S24_3LE
Playing WAVE '/home/marcosch/8_Channel_ID_long.wav' : Signed 24 bit Little Endian in 3bytes, Rate 48000 Hz, Channels 8
IEC958 subframe conversion PCM (IEC958_SUBFRAME_LE)
Its setup is:
stream : PLAYBACK
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 8
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 2048
period_size : 256
period_time : 5333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 256
period_event : 0
start_threshold : 2048
stop_threshold : 2048
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
Slave: Plug PCM: Hardware PCM card 0 'vc4-hdmi-0' device 0 subdevice 0
Its setup is:
stream : PLAYBACK
access : MMAP_INTERLEAVED
format : IEC958_SUBFRAME_LE
subformat : STD
channels : 8
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 2048
period_size : 256
period_time : 5333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 256
period_event : 0
start_threshold : 2048
stop_threshold : 2048
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
appl_ptr : 0
hw_ptr : 0
underrun!!! (at least 48.132 ms long)
Status:
state : XRUN
trigger_time: 2442.383144
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 15.864 ms long)
Status:
state : XRUN
trigger_time: 2443.571155
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 114.361 ms long)
Status:
state : XRUN
trigger_time: 2458.155994
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 86.252 ms long)
Status:
state : XRUN
trigger_time: 2458.502353
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 76.851 ms long)
Status:
state : XRUN
trigger_time: 2458.816363
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 62.822 ms long)
Status:
state : XRUN
trigger_time: 2572.786523
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
However, i was not playing close attention to the audio and i cannot tell if channel switching happened every time there was an underrun.
Tried to repeat the test, but so far, after ca. half dozen trials, even lowering the period size and buffer size, i did not experience underruns anymore... any suggestions on how to force them or is the information above good enough?
@HenrikEnquist you mention the possibility of using snd_pcm_reset() is this something that needs to be implemented in the software of is it something that i could use myself somehow? (sorry for the no clue noob question)
I tested your suggestion:
The first time, i had several buffer underruns, this is what was reported:
marcosch@raspcamilla:~$ aplay -c 8 -t wav -r 48000 -f S24_LE --period-size=256 --buffer-size=2048 -v -D iec958:vc4hdmi0 ~/8_Channel_ID_long.wav
Warning: format is changed to S24_3LE
Playing WAVE '/home/marcosch/8_Channel_ID_long.wav' : Signed 24 bit Little Endian in 3bytes, Rate 48000 Hz, Channels 8
IEC958 subframe conversion PCM (IEC958_SUBFRAME_LE)
Its setup is:
stream : PLAYBACK
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 8
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 2048
period_size : 256
period_time : 5333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 256
period_event : 0
start_threshold : 2048
stop_threshold : 2048
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
Slave: Plug PCM: Hardware PCM card 0 'vc4-hdmi-0' device 0 subdevice 0
Its setup is:
stream : PLAYBACK
access : MMAP_INTERLEAVED
format : IEC958_SUBFRAME_LE
subformat : STD
channels : 8
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 2048
period_size : 256
period_time : 5333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 256
period_event : 0
start_threshold : 2048
stop_threshold : 2048
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
appl_ptr : 0
hw_ptr : 0
underrun!!! (at least 48.132 ms long)
Status:
state : XRUN
trigger_time: 2442.383144
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 15.864 ms long)
Status:
state : XRUN
trigger_time: 2443.571155
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 114.361 ms long)
Status:
state : XRUN
trigger_time: 2458.155994
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 86.252 ms long)
Status:
state : XRUN
trigger_time: 2458.502353
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 76.851 ms long)
Status:
state : XRUN
trigger_time: 2458.816363
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
underrun!!! (at least 62.822 ms long)
Status:
state : XRUN
trigger_time: 2572.786523
tstamp : 0.000000
delay : 0
avail : 2049
avail_max : 2049
However, i was not playing close attention to the audio and i cannot tell if channel switching happened every time there was an underrun.
Tried to repeat the test, but so far, after ca. half dozen trials, even lowering the period size and buffer size, i did not experience underruns anymore... any suggestions on how to force them or is the information above good enough?
@HenrikEnquist you mention the possibility of using snd_pcm_reset() is this something that needs to be implemented in the software of is it something that i could use myself somehow? (sorry for the no clue noob question)
Thanks for the test. Behavior of the channels during the xrun recovery was the reason for the test, so it's really crucial.
As of introducing the xruns, there could be (at least) two reasons - either your CPU load of the respective cores was higher - you can try to increase your CPU load during the test, e.g. using the stress -c X command (see man stress). Or the CPU load was small and CPU cores were switching frequency which causes glitches in the CPUs - see e.g. https://audiosciencereview.com/forum/index.php?threads/rpi4-camilladsp-tutorial.29656/post-1513038
As of introducing the xruns, there could be (at least) two reasons - either your CPU load of the respective cores was higher - you can try to increase your CPU load during the test, e.g. using the stress -c X command (see man stress). Or the CPU load was small and CPU cores were switching frequency which causes glitches in the CPUs - see e.g. https://audiosciencereview.com/forum/index.php?threads/rpi4-camilladsp-tutorial.29656/post-1513038
Sorry for being slightly off-topic with this question: May I ask for a recommendation for an ARM SBC which is more, or even better, completely open? Is e.g. the Odroid C4 more open than the RPi's? Indended use: CDSP of course.
Maybe another way to debug the channels swap item would be to pipe camilladsp | aplay, letting aplay completely handle the hw output part?
i managed to force underruns: i lowered buffer size and period size significantly as you can see below and run in parallel sudo "stress --cpu 2 --timeout 10" a few times.
Most of the underruns did not cause channel switch, but some yes. It is difficult to tell exactly which ones did and which ones didn't because sometimes they happen very close in time. See below the output. I can confirm that the one at trigger_time: 1531.517938 caused a channel switch.
When finished, started camilladsp again with the original settings that gave frequent channel switching events (chuncksize 1024, target level 1024 and adjust period 10) and like initially, i see frequetnt channel switching events.
Hope it helps guys, from my side i regard the issue not solved but under control, as i think i can manage to stay free of underruns. But whatever you want me to try, i will happily do it. Thanks a lot!!
marcosch@raspcamilla:~$ aplay -c 8 -t wav -r 48000 -f S24_3LE --period-size=64 --buffer-size=128 -v -D iec958:vc4hdmi0 ~/8_Channel_ID_long.wav
Playing WAVE '/home/marcosch/8_Channel_ID_long.wav' : Signed 24 bit Little Endian in 3bytes, Rate 48000 Hz, Channels 8
IEC958 subframe conversion PCM (IEC958_SUBFRAME_LE)
Its setup is:
stream : PLAYBACK
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 8
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 128
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 64
period_event : 0
start_threshold : 128
stop_threshold : 128
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
Slave: Plug PCM: Hardware PCM card 0 'vc4-hdmi-0' device 0 subdevice 0
Its setup is:
stream : PLAYBACK
access : MMAP_INTERLEAVED
format : IEC958_SUBFRAME_LE
subformat : STD
channels : 8
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 128
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 64
period_event : 0
start_threshold : 128
stop_threshold : 128
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
appl_ptr : 0
hw_ptr : 0
underrun!!! (at least 0.293 ms long)
Status:
state : XRUN
trigger_time: 1231.518079
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.075 ms long)
Status:
state : XRUN
trigger_time: 1319.602099
tstamp : 0.000000
delay : 0
avail : 65
avail_max : 129
underrun!!! (at least 2.728 ms long)
Status:
state : XRUN
trigger_time: 1320.594446
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.038 ms long)
Status:
state : XRUN
trigger_time: 1348.587835
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.151 ms long)
Status:
state : XRUN
trigger_time: 1350.615235
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.018 ms long)
Status:
state : XRUN
trigger_time: 1356.588114
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.089 ms long)
Status:
state : XRUN
trigger_time: 1396.519126
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.395 ms long)
Status:
state : XRUN
trigger_time: 1448.589557
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.575 ms long)
Status:
state : XRUN
trigger_time: 1451.589065
tstamp : 0.000000
delay : 0
avail : 65
avail_max : 129
underrun!!! (at least 0.032 ms long)
Status:
state : XRUN
trigger_time: 1460.598091
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.316 ms long)
Status:
state : XRUN
trigger_time: 1466.596040
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.240 ms long)
Status:
state : XRUN
trigger_time: 1478.598149
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.165 ms long)
Status:
state : XRUN
trigger_time: 1480.591009
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.030 ms long)
Status:
state : XRUN
trigger_time: 1531.517938
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.031 ms long)
Status:
state : XRUN
trigger_time: 1656.518950
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.189 ms long)
Status:
state : XRUN
trigger_time: 1744.592170
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.096 ms long)
Status:
state : XRUN
trigger_time: 1746.592070
tstamp : 0.000000
delay : 0
avail : 129
avail_max : 129
underrun!!! (at least 0.288 ms long)
Status:
state : XRUN
trigger_time: 1815.601420
tstamp : 0.000000
delay : 0
avail : 65
avail_max : 129
my conclusion as of now after running several tests is clear: not all the underruns cause a channel swap, but all the channel swaps are caused by (or happen simultaneouly to) an underrun
Looking at the RPi hdmi audio driver, it seems to handle xruns properly - sending stop message to VC4 and going to prepare. The key parts were provided by an external developer https://blog.dowhile0.org/2013/04/27/mmap-support-for-raspberry-pi-bcm2835-alsa-driver/ https://github.com/raspberrypi/linux/pull/286
Not much has changed in those driver files recently, it's mostly compatibility maintenance https://github.com/raspberrypi/linux/commits/rpi-6.5.y/drivers/staging/vc04_services/bcm2835-audio
IMO unless the issue is confirmed with an AVR, the RPi devs will blame the HDMI audio extractor (which may be the case).
Not much has changed in those driver files recently, it's mostly compatibility maintenance https://github.com/raspberrypi/linux/commits/rpi-6.5.y/drivers/staging/vc04_services/bcm2835-audio
IMO unless the issue is confirmed with an AVR, the RPi devs will blame the HDMI audio extractor (which may be the case).
thanks @phofman
unfortunately i don't have an AVR to test, but will keep this at the back of my mind if one day i have the chance. So far the two devices i was able to test and show this behaviour are based on the same hdmi interface chip (semiconn EP92A3E) that is commonplace in hdmi extractors.
I also read that ubuntu server 23.04 comes with a newer kernel (i am using 22.04), i could give that a try as per Henrik comments.
Thanks all again for all the help and learnings!
unfortunately i don't have an AVR to test, but will keep this at the back of my mind if one day i have the chance. So far the two devices i was able to test and show this behaviour are based on the same hdmi interface chip (semiconn EP92A3E) that is commonplace in hdmi extractors.
I also read that ubuntu server 23.04 comes with a newer kernel (i am using 22.04), i could give that a try as per Henrik comments.
Thanks all again for all the help and learnings!