I progress slowly 
So as said above, I pulled a stm32F746g discovery board I had never used, but which happens to have a USB HS periph. I ported the only available (at least that I found) stm32/threadX USB device UAC2 example, and got it to work with the embedded CODEC.
This took me time as:
I could get the threadX USB device up fast. It seems well designed. I had some preliminary tests in 5.1 config, and I was receiving data. Stitching the ThreadX USB to the WM8994 Codec took me more time and retro-engineering.
The USB HS => CODEC is now working in stereo (start humble small padawan), and I would have liked to jusmp to multi-channel, but I have some issue with Volume and Mute controls. If I change the volume too fast, the USB thread stops waiting for a Semaphore and Audio stops. If slow... looks OK
I had not tried to measure, but the 2-3ms taken for I2C to write CODEC registers may be part of the issue. That writing time looked a lot to me initially but the board is limited to 100kHz, and Volume was 5 registers to modify 16 bits addresses and values => 160 bits + I2C overhead, so maybe OK. A few requests spread in time seem OK but too much seem to make something fail somewhere. I had not measured in my other projects, but situation should be the same as the same "blocking mode" is used in the different projects I have.
At least, I learn...
JMF
So as said above, I pulled a stm32F746g discovery board I had never used, but which happens to have a USB HS periph. I ported the only available (at least that I found) stm32/threadX USB device UAC2 example, and got it to work with the embedded CODEC.
This took me time as:
- I work for my projects with stm32CubeIDE for low level code generation,
- ST examples are often without .IOC (and was the case for the Azure ThreadX and the discovery stand alone project).
I could get the threadX USB device up fast. It seems well designed. I had some preliminary tests in 5.1 config, and I was receiving data. Stitching the ThreadX USB to the WM8994 Codec took me more time and retro-engineering.
The USB HS => CODEC is now working in stereo (start humble small padawan), and I would have liked to jusmp to multi-channel, but I have some issue with Volume and Mute controls. If I change the volume too fast, the USB thread stops waiting for a Semaphore and Audio stops. If slow... looks OK
I had not tried to measure, but the 2-3ms taken for I2C to write CODEC registers may be part of the issue. That writing time looked a lot to me initially but the board is limited to 100kHz, and Volume was 5 registers to modify 16 bits addresses and values => 160 bits + I2C overhead, so maybe OK. A few requests spread in time seem OK but too much seem to make something fail somewhere. I had not measured in my other projects, but situation should be the same as the same "blocking mode" is used in the different projects I have.
At least, I learn...
JMF
I have a question. How is behaving the host that is sending music frames when the device is blocked by other things (like blocking I2C writing for volume or Mute)?
Is it still sending frames that will be "lost" by the device, with raised low priority interrupts also lost by the device. Or does it stops, waiting of a sort of Ack. And resumes when it gets the green flag?
JMF
Is it still sending frames that will be "lost" by the device, with raised low priority interrupts also lost by the device. Or does it stops, waiting of a sort of Ack. And resumes when it gets the green flag?
JMF
Thanks bohrok2610 for answering the question. It is important to know that the host is not aware that the device is "blocked". I will instrument the IsoOutIncomplete code to learn when it is activated, and if it is triggered in my case.
For the I2C using a non blocking process and/or a lower priority thread would help. I have to check the different options to see what would be possible and robust. This CODEC needs 2 to 4 registers to be written in a consistent way. If the CPU is loaded and the task is not activated often, I have to be carefull. I have to see if the registers are one after the other and can be written with a single DMA request.
Or see it can stay as is but not generate a deadlock... loosing few frames on Mute ot Volume changes can be acceptable/
For the I2C using a non blocking process and/or a lower priority thread would help. I have to check the different options to see what would be possible and robust. This CODEC needs 2 to 4 registers to be written in a consistent way. If the CPU is loaded and the task is not activated often, I have to be carefull. I have to see if the registers are one after the other and can be written with a single DMA request.
Or see it can stay as is but not generate a deadlock... loosing few frames on Mute ot Volume changes can be acceptable/
I'm slowly progressing on the project. I implemented the Mute and Volume control in lower priority thread using I2C non blocking write procedures. This works much better. It's now time to revert to the multi-channel tests.
At first, I DON'T want to use TDM.
I was wondering if some SAI features could Help "demultiplex" the USB frame to multiple SAI outputs (I2S) format, using the Slot features in a 8 channels IN SAI, 2 OUT (for the 2 selected slots). It seems to me that it is not the case: SAI only "switch off" the data line for the Inactive Slot: 8 In - 8 Out (with some slots "switched-off".
Is this understanding correct ? So needs to split the usb delivered buffer in independent buffers per pair of channels and feed those in the SAI.
Best regards,
JMF
At first, I DON'T want to use TDM.
I was wondering if some SAI features could Help "demultiplex" the USB frame to multiple SAI outputs (I2S) format, using the Slot features in a 8 channels IN SAI, 2 OUT (for the 2 selected slots). It seems to me that it is not the case: SAI only "switch off" the data line for the Inactive Slot: 8 In - 8 Out (with some slots "switched-off".
Is this understanding correct ? So needs to split the usb delivered buffer in independent buffers per pair of channels and feed those in the SAI.
Best regards,
JMF
So I have now 6 channels 16 bits 48k code working, based on ThreadX USBX stack. Took time, but it is as much to learn than to deliver. That's OK.
This uses about 50% of the CPU of a stm32F746 clocked at 200Mz. I have to do some experiments with 8 channels, 24 bits and increased sampling rate.
And I need to understand how the Async feedback is intended to be implemented in that stack. Elements seems in place (conditionally included code for creation of EP feedback, procedures), but no working example on the net (at least that I found)
JMF
This uses about 50% of the CPU of a stm32F746 clocked at 200Mz. I have to do some experiments with 8 channels, 24 bits and increased sampling rate.
And I need to understand how the Async feedback is intended to be implemented in that stack. Elements seems in place (conditionally included code for creation of EP feedback, procedures), but no working example on the net (at least that I found)
JMF
Last edited:
I digged a bit in the USBX implementation of the feedback. I don't understand all the code, it it seems to take care of the creation of the end points, and the transfer of the feedback to the Host. I don't seem to take care of the evaluation of the value to send. I looked at the USB 2.0 spec, to better understand what is to be sent to the Host.
My idea is, at the end of the DMA transfer to SAI (HAL_DMA_IRQHandler call back ; DMA in circular buffer), to assess the position of the write buffer compared to the middle of the buffer (time at which I started triggering the DMA). The gap is then converted to "transfert rate" and encoded in 16.16 format and given to USBX. My buffer is 2ms (96 samples at 48kHz).
Any obvious flaw in this approach ?
The en.x-cube-usb-audio code is very basic: adjust the target sampling frequence by +/-1000Hz depending of being below or above the reference level of the buffer (Bang Bang control).
JMF
My idea is, at the end of the DMA transfer to SAI (HAL_DMA_IRQHandler call back ; DMA in circular buffer), to assess the position of the write buffer compared to the middle of the buffer (time at which I started triggering the DMA). The gap is then converted to "transfert rate" and encoded in 16.16 format and given to USBX. My buffer is 2ms (96 samples at 48kHz).
Any obvious flaw in this approach ?
The en.x-cube-usb-audio code is very basic: adjust the target sampling frequence by +/-1000Hz depending of being below or above the reference level of the buffer (Bang Bang control).
JMF
@JMF11: The linux UAC1/2 gadget sources contain quite a bit of know-how about what Win/OSX/Linux OS drivers support. May be handy for your reference, e.g. https://github.com/torvalds/linux/b...96/drivers/usb/gadget/function/u_audio.c#L100
That linked method generates the feedback value. Important info - the pitch is rate correction ratio, 1000000 means 1, no correction. That value is provided by user-space programs (because the gadget driver itself does no playback, i.e. has no clock to compare with to generate info about the required rate adjustments).
I'm working on tyhe Async feedback mode on the Azure USBX stack. I started from the now working 8 channels code and started to add the USBX code and thread related to the Async feedback, including now the modification of the descriptor to include the feedback specificities. The code to provide a proper value for the feedback is not implemented yet. The device is recognized under Linux (good), but not under Windows...
:
But I have an error under windows... The driver gets a yellow triangle. The UAC2 log is attached below. It seems to fail in the MiniportWave part
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]MiniportWave: FFFFBE8F7C21EC80
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]returning 0xFFFFBE8F7D6F15A0
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]port->Init() failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]CreateSubDevice wave failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]CreateFilterGraph failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]StartAdapter() failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.2860::04/27/2024-22:18:00.556 [USBAudio2][ 0x02 <NULL>
Bus 007 Device 002: ID 0483:5730 STMicroelectronics Audio Speaker
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x0483 STMicroelectronics
idProduct 0x5730 Audio Speaker
bcdDevice 2.00
iManufacturer 1 STMicroelectronics
iProduct 2 STM32 AUDIO20 STREAMING
iSerial 3 AUDIO001
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0098
bNumInterfaces 2
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xc0
Self Powered
MaxPower 50mA
Interface Association:
bLength 8
bDescriptorType 11
bFirstInterface 0
bInterfaceCount 2
bFunctionClass 1 Audio
bFunctionSubClass 0
bFunctionProtocol 32
iFunction 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 1 Audio
bInterfaceSubClass 1 Control Device
bInterfaceProtocol 32
iInterface 0
AudioControl Interface Descriptor:
bLength 9
bDescriptorType 36
bDescriptorSubtype 1 (HEADER)
bcdADC 2.00
bCategory 1
wTotalLength 0x0040
bmControls 0x00
AudioControl Interface Descriptor:
bLength 8
bDescriptorType 36
bDescriptorSubtype 10 (CLOCK_SOURCE)
bClockID 24
bmAttributes 1 Internal fixed clock
bmControls 0x01
Clock Frequency Control (read-only)
bAssocTerminal 0
iClockSource 0
AudioControl Interface Descriptor:
bLength 17
bDescriptorType 36
bDescriptorSubtype 2 (INPUT_TERMINAL)
bTerminalID 18
wTerminalType 0x0101 USB Streaming
bAssocTerminal 0
bCSourceID 24
bNrChannels 8
bmChannelConfig 0x00000003
Front Left (FL)
Front Right (FR)
iChannelNames 0
bmControls 0x0000
iTerminal 0
AudioControl Interface Descriptor:
bLength 18
bDescriptorType 36
bDescriptorSubtype 6 (FEATURE_UNIT)
bUnitID 22
bSourceID 18
bmaControls(0) 0x0000000f
Mute Control (read/write)
Volume Control (read/write)
bmaControls(1) 0x00000000
bmaControls(2) 0x00000000
iFeature 0
AudioControl Interface Descriptor:
bLength 12
bDescriptorType 36
bDescriptorSubtype 3 (OUTPUT_TERMINAL)
bTerminalID 20
wTerminalType 0x0301 Speaker
bAssocTerminal 0
bSourceID 22
bCSourceID 24
bmControls 0x0000
iTerminal 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 1 Audio
bInterfaceSubClass 2 Streaming
bInterfaceProtocol 32
iInterface 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 1
bNumEndpoints 2
bInterfaceClass 1 Audio
bInterfaceSubClass 2 Streaming
bInterfaceProtocol 32
iInterface 0
AudioStreaming Interface Descriptor:
bLength 16
bDescriptorType 36
bDescriptorSubtype 1 (AS_GENERAL)
bTerminalLink 18
bmControls 0x00
bFormatType 1
bmFormats 0x00000001
PCM
bNrChannels 8
bmChannelConfig 0x000000ff
Front Left (FL)
Front Right (FR)
Front Center (FC)
Low Frequency Effects (LFE)
Back Left (BL)
Back Right (BR)
Front Left of Center (FLC)
Front Right of Center (FRC)
iChannelNames 0
AudioStreaming Interface Descriptor:
bLength 6
bDescriptorType 36
bDescriptorSubtype 2 (FORMAT_TYPE)
bFormatType 1 (FORMAT_TYPE_I)
bSubslotSize 2
bBitResolution 16
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 5
Transfer Type Isochronous
Synch Type Asynchronous
Usage Type Data
wMaxPacketSize 0x0060 1x 96 bytes
bInterval 1
AudioStreaming Endpoint Descriptor:
bLength 8
bDescriptorType 37
bDescriptorSubtype 1 (EP_GENERAL)
bmAttributes 0x00
bmControls 0x00
bLockDelayUnits 0 Undefined
wLockDelay 0x0000
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 17
Transfer Type Isochronous
Synch Type None
Usage Type Feedback
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 4
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x0483 STMicroelectronics
idProduct 0x5730 Audio Speaker
bcdDevice 2.00
iManufacturer 1 STMicroelectronics
iProduct 2 STM32 AUDIO20 STREAMING
iSerial 3 AUDIO001
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0098
bNumInterfaces 2
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xc0
Self Powered
MaxPower 50mA
Interface Association:
bLength 8
bDescriptorType 11
bFirstInterface 0
bInterfaceCount 2
bFunctionClass 1 Audio
bFunctionSubClass 0
bFunctionProtocol 32
iFunction 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 1 Audio
bInterfaceSubClass 1 Control Device
bInterfaceProtocol 32
iInterface 0
AudioControl Interface Descriptor:
bLength 9
bDescriptorType 36
bDescriptorSubtype 1 (HEADER)
bcdADC 2.00
bCategory 1
wTotalLength 0x0040
bmControls 0x00
AudioControl Interface Descriptor:
bLength 8
bDescriptorType 36
bDescriptorSubtype 10 (CLOCK_SOURCE)
bClockID 24
bmAttributes 1 Internal fixed clock
bmControls 0x01
Clock Frequency Control (read-only)
bAssocTerminal 0
iClockSource 0
AudioControl Interface Descriptor:
bLength 17
bDescriptorType 36
bDescriptorSubtype 2 (INPUT_TERMINAL)
bTerminalID 18
wTerminalType 0x0101 USB Streaming
bAssocTerminal 0
bCSourceID 24
bNrChannels 8
bmChannelConfig 0x00000003
Front Left (FL)
Front Right (FR)
iChannelNames 0
bmControls 0x0000
iTerminal 0
AudioControl Interface Descriptor:
bLength 18
bDescriptorType 36
bDescriptorSubtype 6 (FEATURE_UNIT)
bUnitID 22
bSourceID 18
bmaControls(0) 0x0000000f
Mute Control (read/write)
Volume Control (read/write)
bmaControls(1) 0x00000000
bmaControls(2) 0x00000000
iFeature 0
AudioControl Interface Descriptor:
bLength 12
bDescriptorType 36
bDescriptorSubtype 3 (OUTPUT_TERMINAL)
bTerminalID 20
wTerminalType 0x0301 Speaker
bAssocTerminal 0
bSourceID 22
bCSourceID 24
bmControls 0x0000
iTerminal 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 1 Audio
bInterfaceSubClass 2 Streaming
bInterfaceProtocol 32
iInterface 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 1
bNumEndpoints 2
bInterfaceClass 1 Audio
bInterfaceSubClass 2 Streaming
bInterfaceProtocol 32
iInterface 0
AudioStreaming Interface Descriptor:
bLength 16
bDescriptorType 36
bDescriptorSubtype 1 (AS_GENERAL)
bTerminalLink 18
bmControls 0x00
bFormatType 1
bmFormats 0x00000001
PCM
bNrChannels 8
bmChannelConfig 0x000000ff
Front Left (FL)
Front Right (FR)
Front Center (FC)
Low Frequency Effects (LFE)
Back Left (BL)
Back Right (BR)
Front Left of Center (FLC)
Front Right of Center (FRC)
iChannelNames 0
AudioStreaming Interface Descriptor:
bLength 6
bDescriptorType 36
bDescriptorSubtype 2 (FORMAT_TYPE)
bFormatType 1 (FORMAT_TYPE_I)
bSubslotSize 2
bBitResolution 16
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 5
Transfer Type Isochronous
Synch Type Asynchronous
Usage Type Data
wMaxPacketSize 0x0060 1x 96 bytes
bInterval 1
AudioStreaming Endpoint Descriptor:
bLength 8
bDescriptorType 37
bDescriptorSubtype 1 (EP_GENERAL)
bmAttributes 0x00
bmControls 0x00
bLockDelayUnits 0 Undefined
wLockDelay 0x0000
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 17
Transfer Type Isochronous
Synch Type None
Usage Type Feedback
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 4
But I have an error under windows... The driver gets a yellow triangle. The UAC2 log is attached below. It seems to fail in the MiniportWave part
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]MiniportWave: FFFFBE8F7C21EC80
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]returning 0xFFFFBE8F7D6F15A0
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]port->Init() failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]CreateSubDevice wave failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]CreateFilterGraph failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.3B24::04/27/2024-22:18:00.556 [USBAudio2]StartAdapter() failed 0xc000028c(STATUS_RANGE_NOT_FOUND)
[0]0004.2860::04/27/2024-22:18:00.556 [USBAudio2][ 0x02 <NULL>
The Windows driver was blocked by an unsufficient size for the EP01 wMaxPacketSize. It was missing the space for the extra sample in the packet to allow for the feedback mecanism. With that size adjusted, the device is recognized and I can stream music.
The strange thing is that the feedback task is only triggered if interval is set to 0x01. When using bInterval = 0x02 or 0x04, I don't see anymore the feedback task activated anymore during the streaming :-( . This mqy be consistent with https://learn.microsoft.com/en-us/a...cit-feedback-for-asynchronous-audio-with-usba, but not aligned with other codes like https://github.com/nxp-mcuxpresso/m...speaker/bm/cm33_core0/usb_device_descriptor.h or https://www.diyaudio.com/community/threads/open-sourced-uac2-bridge-based-on-stm32.404656/
Both above use a 0x04 interval
Could be related to this parity checks to be done?
And I need to implement the feedback value assessment based on buffer level.
The strange thing is that the feedback task is only triggered if interval is set to 0x01. When using bInterval = 0x02 or 0x04, I don't see anymore the feedback task activated anymore during the streaming :-( . This mqy be consistent with https://learn.microsoft.com/en-us/a...cit-feedback-for-asynchronous-audio-with-usba, but not aligned with other codes like https://github.com/nxp-mcuxpresso/m...speaker/bm/cm33_core0/usb_device_descriptor.h or https://www.diyaudio.com/community/threads/open-sourced-uac2-bridge-based-on-stm32.404656/
Both above use a 0x04 interval
Could be related to this parity checks to be done?
And I need to implement the feedback value assessment based on buffer level.
Yes https://github.com/torvalds/linux/b...a96/drivers/usb/gadget/function/f_uac2.c#L702
If by the feedback task you mean activity on the host/windows side, then - do you have the feedback value calculated per microframe time (125us) or per real packet (i.e. bInterval involved)? https://github.com/torvalds/linux/b...ivers/usb/gadget/function/u_audio.c#L136-L138
I understand from the UAC2.0 spec that the feedback value should reflect the expected number of samples in a microframe. For audio bInterval = 1 => a frame every 125uS, 48K sampling rate => something around 6 samples per uFrame, in 16.16 format (around 0x00060000). And this even if the bInterval for feedback is 1, or 2, or 4...
Well specs are one thing, but the linux UAC2 gadget does what the comment says and Win10 async feedback works for any (reasonable) bInterval. Look at the discussion https://lore.kernel.org/all/9f719bb8-2a9a-4a97-f25a-660d38a29555@ivitera.com/t/
Of course I am talking about the data EP OUT bInterval, not about the sync feedback bInterval - here may be the misunderstanding
That is fixed to 4 in the gadget code.
Of course I am talking about the data EP OUT bInterval, not about the sync feedback bInterval - here may be the misunderstanding
That is fixed to 4 in the gadget code.