synchonized streaming to multiple clients/loudspeakers

Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.
I wanted to start a thread on this topic. I'm curious if people know of software that can keep two or more streams very tightly synchronized. I do not mean "multiroom streaming" - that just needs to be approximately synchronized, e.g. below 1 second difference. I am talking about 1 millisecond (!) or less difference between the playback on two or more clients.

Why have such a strict requirement for synchronicity? For example, it is to send audio separately to playback systems in the left speaker and the right speaker. In order for the stereo image to stay in the middle and not wander from side to side, the time difference must stay well below 1 msec.

Another reason I am asking is that I just made some changes to the Gstreamer based app I wrote and that I use to stream audio around my home. I now seem to be able to stream to multiple clients and keep the stereo image centered. This means that I will be able to build two completely self-contained, independent active speakers and deploy them as a left+right pair anywhere in my home, with just an AC mains connection/plug and no other external wiring.

If there is enough interest in this from others I will be happy to share my app with everyone. It's a much improved version of my GSASysCon app that I first described a few years ago. It can stream and implement DSP on the clients. Add multichannel DACs at the clients and you have a streaming, DSP crossover audio system that can be reconfigured at will. In the past, it has been the documentation that has prevented me from releasing versions as I make improvements because it is very tedious and complicated to explain everything in sufficient detail. It would be great if I could get some "testers" signed up to do some test runs and give me feedback on everything.

NOTE: when I say "app" I do not mean something that you install on your phone. This is a Linux based bash script with a Gstreamer backend doing all the heavy lifting. You must be able to have multiple Linux computers (R-Pi 3 or 4 or similar, minimum) and DACs to implement this system. Experience setting up and maintaining headless systems and using SSH is useful.
 
I'm planning on using Ravenna by Merging for this. But it doesn't support WiFi and it claims 5ms sync. It does support DSD, though.
I'd be curious to see how you maintain sub ms clock sync over WiFi.

5msec will disappoint you.
Ravenna has a pro audio streaming platform but it requires PTP hardware support, which is not supported over WiFi.

Gstreamer is the engine that handles all the streaming, routing, DSP, etc. It uses RTP+RTCP to keep synchronization very tight. My code is built around it. I have the computers synchronized via NTP to a single NTP server computer I set up in my home using a spare linux box. This keeps all clocks within about 100 usec or less of each other. I am about to test a non-NTP synchronized R-Pi based 2 client system to see how that goes.

More info in this presentation:
Synchronised multi-room media playback and distributed live media processing and mixing with GStreamer - GStreamer conferences
 
Last edited:
Gstreamer has a built in NTP-like time synchronization mechanism that is used to bring all pipeline clocks (on sender and receivers) onto the same time basis.

From the presentation in the last post, slide 13:

GStreamer Net & NTPv4 Clock
  • Gstreamer's NetClock existing since around 2005
  • Uses a custom protocol similar to NTPv4
  • UDP messages sent between client and server:
    1. Client “asks” the server for the current time & estimates the round-trip-time
    2. Uses both the estimate the current time at packet receival
  • NTPv4 (RFC4330) client clock since version 1.6 – Shares most of the code with the NetClock
  • Tricky part: Filtering to handle networks with fluctuating RTTs
 
I have started to create a new, independent group of three Raspberry Pi 4B computers to test out the synchronization in the absence of NTP.

One R-Pi will act as the streaming audio server. It will have an audio input, either spdif or analog (have not decided yet). It serves up a WiFi network to which the clients can connect.

The two client computers will each be incorporated onto the back panel of a small 2-way speaker that I built several years ago. Its an active DSP speaker so there is no existing onboard crossover - that was previous done via outboard components. To implement the crossover and amplification I am using the DSP crossover functionality of my GSASysCon software. For amplification I am using HiFiBerry Amp2 HATs plus a Meanwell LRS100-24 SMPS. These components can all be mounted to the back of the speaker, with an AC power cord providing power to the system. There will be no other physical wiring to the speaker. I am still in the process of setting up the clients, but hopefully I should be able to get them working in the next few days.

This implementation is portable because the server generates its own WiFi network. I could take it to another location like a DIY show and it should work without having to do any changes or setup. I can add more clients, e.g. for a subwoofer, etc. as necessary.
 
UPDATE:

I finally have the multi-client synchronized streaming test bed up and running, the one described in the last post. It's alive and working well!

I have been streaming 24bit uncompressed audio wirelessly over a private 802.11n network and so far everything seems to work perfectly.

I've attached a pic of the SMPS and Pi+Amp2 HAT bolted on to the back of the speaker. Sorry the lighting was bad and I snapped this with my phone, so the pic quality isnt' all that great.

I will do more listening and testing over the next few days.
 

Attachments

  • IMG_20200817_214800121.jpg
    IMG_20200817_214800121.jpg
    106 KB · Views: 394
I have done several hours of listening now and the system seems to be working great. Each speaker is running an independent client Pi and the DSP crossover filters are implemented on the clients, in software. While streaming 16/48, 24/48, or 24/96 uncompressed audio the speakers never lost sync and I did not hear any audible glitches or other hiccups. The system can be started from a "power off" state to active playback in about 1 minute.

This is a nice, standalone system when set up this way. But I can also connect the "server" Pi via its ethernet port to my home network and source the audio that way. The onboard WiFi is being used to serve up the 802.11n network for the clients, leaving only the ethernet port for connection to another network leading to the greater internet. I suppose that I could use a WiFi extender box that I own to access my home's WiFi network by using it in access point mode.

I would also like to try to set up the "server" Pi as a DLNA endpoint or Airplay client, etc. so that I could stream audio from my phone to the speaker system.
 
Below are a couple more pics of the system, set up in my basement speaker testing cave: front side and back side of the speaker, and a shot of the Pi with ADC HAT connected to CD player input.

The system is working great and is small enough to be installed right on the speaker. Amp HAT puts out about 30-40 WPC, 2 channels, in a nice and compact package. Temps on the Pis and the amp HAT seem to be quite reasonable, although I have not really pushed the amp to high output levels yet. The Pis are set for a fixed CPU frequency of 1GHz and with no heatsinking run at around 55-60C depending on the ambient temp. I also reduced the CPU core voltage a few notches to keep the heat generation down. The CPU usage on the clients, while receiving the audio stream and doing DSP processing, is only around 35% of one core (on a Pi 4), so the Pi is not really working that hard.

attachment.php


attachment.php


attachment.php
 

Attachments

  • IMG_20200819_110713829.jpg
    IMG_20200819_110713829.jpg
    99.4 KB · Views: 948
  • IMG_20200819_110733735.jpg
    IMG_20200819_110733735.jpg
    123.5 KB · Views: 646
  • IMG_20200819_110855586.jpg
    IMG_20200819_110855586.jpg
    129.3 KB · Views: 631
I've made some improvements and modifications to the setup and code. I have (in my estimation) reduced the total end-to-end latency down to around 100 msec. This is not good enough for e.g. using as the audio that comes with an AV stream e.g. movies, but for pure audio listening this is very usable and responds in nearly real time to volume changes, etc. Since the total latency includes ALSA buffering on both ends, plus the latency of sending the audio and doing DSP processing on the clients, I think that 100msec is not bad at all. Hey, if you really want zero latency, I suggest that you use a wire! :rolleyes:

The audio playback is faultless as far as I can tell by listening. So far I have primarily been streaming 24 bits at 96kHz, but may try a higher sample rate to see what happens. The stream consists of uncompressed audio samples, so fidelity is high.

For a digital input to the setup, it is possible to employ a miniDSP USBstreamer or MCHstreamer. Both support toslink spdif, and the MCH has a header where you can easily connect coax spdif. There may be HATs for the Pi that also support digital input, however, I only have first hand experience with, and can recommend, the miniDSP based solution.
 
Last edited:
Audio delay is not a problem for movies or TV as long as it's constant. TVs and media players these days can delay the video by a preset time.
Yes, if you can also delay the video signal somehow, that's perfect.

Can you control the latency from start to end?
The total audio latency is not entered as a parameter per se. It's the sum of various contributions along the way. Some of these you can specify with parameters and some are just a consequence of processing delays. But once the streaming system is launched the latency remains unchanged going forward. The user should be able to get a good estimate of the total latency, or it could be measured by doing a kind of "loopback" test through the streaming system.

Also, does gstreamer support DSD?
I think the answer to the question you are trying to ask is: no. The hangup is not Gstreamer per se (a large set of AV processing related utilities), but rather the RTP data transfer protocol that is used to send the audio data plus playback timing information over the LAN/WiFi. According to this Wikipedia page on RTP payload types DSD is not supported:
RTP payload formats - Wikipedia
Also, DSD support seems to be spotty under Linux from what I can tell. I'm not totally sure, since I do not use that format and things are changing/improving all the time.

In any case, the app is not typically used to play files directly (if that was what you were envisioning). Some other "player" software (e.g. DeaDBeef) is used to read/play the DSD file and send the audio to ALSA or PulseAudio (for files and internet streams). Audio is typically obtained from ALSA in a supported ALSA data format (e.g. F32_LE or S32_LE). For live sources connected to the local computer (e.g. analog signal via ADC, or SPDIF digital input), these will also be available from ALSA or PulseAudio. When the audio data is converted into a PCM format it is ensured that it can be payloaded into RTP frames and streamed to the clients.
 
I have been playing around with the system, and using a couple of USB devices that support digital audio input. This permits high quality audio to be streamed thru the system to the speakers without having to use an ADC stage.

So far I have gotten these to work:
miniDSP USBStreamer: This has a toslink input that supports up to 24/192 spdif inputs. In order to get this to work under ALSA I had to plug it in, and then manually edit the file /var/lib/alsa/asound.state. The problem seems to be that the default driver sets the input channel map left and right to point to ALSA channel zero (0), which is an "undefined" channel assignment. I edited the state file so that the left and right inputs point to ALSA channels 3 and 4, which are front-left and front-right. Oddly enough, after a reboot the state file again shows the channel map pointing to channel 0, but when I query the channel map using amixer it shows that the channels are set to 3 and 4 and the audio works. Anyway the change seems to be permanent, so it's a one-time edit and you are all set to go. On the plus side, the USBStreamer ALSA driver includes a volume control for the input level, which is needed when you connect to a source that does not have volume such as a CD player. My streaming audio app can access this control directly as part of the interface, and it is very useful to be able to change the volume from the listening chair so I always try to set up such a control. The USBStreamer costs $95 ($105 in an enclosure) + shipping from miniDSP.

HiFiMeDIY UX1: This DAC also features a digital toslink input that supports audio up to 16/48. It's enough to accept the 16/44.1 digital output of a CD player and it works with ALSA out of the box. One problem is that there is no volume control for the digital input, however, you can easily add one using ALSA's softvol device via your .asoundrc file. I set one up with 60dB range and 1dB increments and it works great. Cost is about US$70+shipping. I was able to get one in hand from a US warehouse on the west coast in a couple of days time.

Both of the above are small, bus-powered devices that support toslink (optical) spdif.

I also attempted to use a HiFiBerry DIGI+ I/O HAT but could not get it to work and returned it.

Upcoming testing:

I also have on hand a Focusrite Scarlett 6i6. In addition to analog inputs it has a coax spdif input and can support up to 24/192. I have not yet had a chance to test it on this system but I hope to do that over the weekend. The 6i6 has been discontinued I believe, but you can still buy a refurbished one from them for under $150, which is a great deal.

I would love to hear about other affordable soundcards (USB or HATs) that can accept/record digital audio under Linux, with kernel 5.4 or later (if that is known).

I will also try to set up the Pi 4 as an OTG device, either an audio device or a ethernet device. This is only possible when powering the Pi 4 via its GPIO pins, because you need to use the USB-C port for OTG and it cannot simultaneously support charging. I now have a HAT that will allow me to plug in my USB-C power supply there, so I am now able to think about the OTG Gadget for audio input.

It will be nice to have different options to pick from for digital audio, plus analog input if that is needed. This forms the basis of a preamp that can stream audio to a nearby active loudspeaker system.
 
Still working on interfaces for the Pi streaming server. In the meantime I am trying to figure out why that system (consisting of three Pi 4Bs) is working so well, while another system that uses three Intel boxes is working poorly. I think the answer may lie in my WiFi setup.

I have a gateway+router that I got from my ISP (Xfinity). It has pretty crappy WiFi and I had to locate it in the basement, so I decided to look for another means to roll out WiFi networking. My home does have some hardwired ethernet cabling installed here and there, so I bought a Netgear WiFi "extender" (EX7000) that I installed on the first floor and which is connected to a wired internet jack. It serves up the WiFi connections throughout the home and coverage is pretty good.

In this application we need a low latency connection, and both from server to client and client to server. This is because there is the audio flowing from server to client, but there is also timing information returning to the server from the clients. If either of the directions has unacceptably long or widely varying latency, things will tend to fall apart.

In addition, when setting up the Pi based system I used hostapd to create the isolated network. The isolated network keeps out any unrelated traffic. As part of the configuration I enabled WMM (quality) based prioritization for the connection. This may be one reason why, even thought he signal level is not high, it performs well. Unfortunately the Netgear WiFi extender based access point I use in my home does not include Qos or WMM features as part of its software. It's a bit old now and sometimes buggy, so I am going to replace it with a Netgear R7800, which has good 5.8G band performance and the Qos and WMM capabilities I need.

As an example of my current wired vs wireless performance, let's look at the Intel streaming audio system I mentioned is not working so well. There are two identical client machines, one set up for the right speaker and one for the left speaker. I moved one from a 5.8G WiFi connection to a wired connection and then pinged it. Here is the result on the wired connection:
Code:
ping XXX.0.0.231
PING XXX.0.0.231 (XXX.0.0.231) 56(84) bytes of data.
64 bytes from XXX.0.0.231: icmp_seq=1 ttl=64 time=0.733 ms
64 bytes from XXX.0.0.231: icmp_seq=2 ttl=64 time=0.746 ms
64 bytes from XXX.0.0.231: icmp_seq=3 ttl=64 time=0.728 ms
64 bytes from XXX.0.0.231: icmp_seq=4 ttl=64 time=0.752 ms
64 bytes from XXX.0.0.231: icmp_seq=5 ttl=64 time=0.758 ms
--- ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4056ms
rtt min/avg/max/mdev = 0.728/0.743/0.758/0.011 ms
This looks pretty good. Latency is low and consistent.

Next, I pinged the other machine that is still using the WiFi connection. Here are the results:
Code:
ping XXX.0.0.232
PING XXX.0.0.232 (XXX.0.0.232) 56(84) bytes of data.
64 bytes from XXX.0.0.232: icmp_seq=1 ttl=64 time=45.0 ms
64 bytes from XXX.0.0.232: icmp_seq=2 ttl=64 time=170 ms
64 bytes from XXX.0.0.232: icmp_seq=3 ttl=64 time=194 ms
64 bytes from XXX.0.0.232: icmp_seq=4 ttl=64 time=11.8 ms
64 bytes from XXX.0.0.232: icmp_seq=5 ttl=64 time=34.7 ms
64 bytes from XXX.0.0.232: icmp_seq=6 ttl=64 time=56.7 ms
64 bytes from XXX.0.0.232: icmp_seq=7 ttl=64 time=83.9 ms
64 bytes from XXX.0.0.232: icmp_seq=8 ttl=64 time=103 ms
--- ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7008ms
rtt min/avg/max/mdev = 11.849/87.357/193.716/60.866 ms
As you can see, the average latency to the WiFi machine is much longer and it is highly variable. Packets of audio data are not going to arrive in a timely manner, and this is probably why I need to set the client's receive buffer latency to something on the order of 500msec for audio to be mostly glitchless. Even then, there are sometimes problems that take awhile to resolve themselves. On the other hand, with a 60msec or 100msec receive buffer on the Pi based system playback is perfect and there are never (or almost never) any glitches in the audio.

To compare, if I ping a client Pi from the server Pi I get latencies around 1.5 to 1.6 msec with very few outliers.

Hopefully my new WiFi router (which I will again configure as an access point) will improve the latency problem on the main WiFi network in my home and will prioritize audio traffic. After it arrives I will follow up with some new info and results as time permits.
 
Last edited:
Member
Joined 2002
Paid Member
hi CharlieLaub,

Just for a comparison, here's my RPi4B Wifi ping test. Nothing special, just using RPi4B in-built wifi to a ISP provided wifi router. Distance 20 meters and a couple of internal walls.

Code:
PING 192.168.1.51 (192.168.1.51): 56 data bytes
64 bytes from 192.168.1.51: seq=0 ttl=64 time=2.692 ms
64 bytes from 192.168.1.51: seq=1 ttl=64 time=2.453 ms
64 bytes from 192.168.1.51: seq=2 ttl=64 time=2.369 ms
64 bytes from 192.168.1.51: seq=3 ttl=64 time=3.497 ms
64 bytes from 192.168.1.51: seq=4 ttl=64 time=2.265 ms
64 bytes from 192.168.1.51: seq=5 ttl=64 time=2.085 ms
64 bytes from 192.168.1.51: seq=6 ttl=64 time=2.901 ms
64 bytes from 192.168.1.51: seq=7 ttl=64 time=2.738 ms
64 bytes from 192.168.1.51: seq=8 ttl=64 time=2.981 ms
64 bytes from 192.168.1.51: seq=9 ttl=64 time=3.504 ms
64 bytes from 192.168.1.51: seq=10 ttl=64 time=18.660 ms
64 bytes from 192.168.1.51: seq=11 ttl=64 time=2.167 ms
64 bytes from 192.168.1.51: seq=12 ttl=64 time=3.021 ms
64 bytes from 192.168.1.51: seq=13 ttl=64 time=3.284 ms
64 bytes from 192.168.1.51: seq=14 ttl=64 time=2.875 ms
64 bytes from 192.168.1.51: seq=15 ttl=64 time=3.771 ms
64 bytes from 192.168.1.51: seq=16 ttl=64 time=2.430 ms
64 bytes from 192.168.1.51: seq=17 ttl=64 time=2.295 ms
64 bytes from 192.168.1.51: seq=18 ttl=64 time=2.274 ms
64 bytes from 192.168.1.51: seq=19 ttl=64 time=2.087 ms

--- 192.168.1.51 ping statistics ---
20 packets transmitted, 20 packets received, 0% packet loss
round-trip min/avg/max = 2.085/3.517/18.660 ms

Probably a major factor: I live by myself, no extra traffic!

regards
 
If you are anyway going to be replacing stuff, I would recommend to take a look at Ubiquiti.
We used to run the ISP provided router together with a second router in AP mode. This was never really working well, always a little slow and unreliable. Last autumn I got fed up and replaced it with a system from Ubiquiti. It's a small Edgerouter X and two Unifi access points. This has been rock solid. This turned out to be very useful since suddenly we started working from home a lot more...
 
Member
Joined 2002
Paid Member
About 20 meters.

Code:
---------------------------------------------------------------------------------------
        Quality   Level       Channel      Encryption       Address
---------------------------------------------------------------------------------------
           72     -59dBm     6 (2.437GHz)   on  WPA2

I usually turn off auto channel selection and pick one not used by the neighbours.
 
If you are anyway going to be replacing stuff, I would recommend to take a look at Ubiquiti.
We used to run the ISP provided router together with a second router in AP mode. This was never really working well, always a little slow and unreliable. Last autumn I got fed up and replaced it with a system from Ubiquiti. It's a small Edgerouter X and two Unifi access points. This has been rock solid. This turned out to be very useful since suddenly we started working from home a lot more...

From what I could tell, they all seem to require PoE. I do not have that in my home.

Anyway, I already purchased the Netgear R7800. It's a model from 2016 but there is lots of testing data that shows it has very good 5.8G band performance. I'm familiar with the Netgear products, and it will be a drop-in replacement for the Netgear access point I own now.
 
Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.