A bash-script-based streaming audio system client controller for gstreamer

wouldn't you want those env vars to be managed on the client as they might be client specific values?

Not sure what you mean by "managed on the client". In the example I gave:
Code:
ssh user@host ARG1="value1" ARG2="value2" /bin/bash << ENDSSH
  # commands to run on remote host
  # ARG1 and ARG2 are now environmental variables within the ssh session on the client
  lines of code that are run on the client go here...
ENDSSH
ARG1 and ARG2 become environmental variables on the client within and for the duration of that ssh session only.
 
At this time everything seems to be working to the point that I can start explaining more about the system setup.

Each system that the user would like to control gets its own directory. There is a configuration file in the directory where the user describes the connections to the system, etc. Here is the prototype for this configuration file that I am currently using to test the code:
Code:
AUDIO_SOURCE=alsasrc device='dsnoop:0,1' #gstreamer source string
AUDIO_SOURCE_CAPS=audio/x-raw,format=S16LE,rate=48000,channels=2
PORT=1234             #port on clients where RTP stream will be directed 
STREAM_BITS=16        #sets bit depth of stream and playback
STREAM_RATE=48000     #sets sample rate of stream and playback
VOLUME_TRIM_DB=0      #used to adjust system volume in 1dB steps
#describe first client in the system
CLIENT=192.168.10.111, mono  #one line per client indicating IP and channel
ACCESS=ssh pi@192.168.10.111 USER="pi" LADSPA_PATH="/usr/local/lib/ladspa:/usr/lib/ladspa" TERM="xterm" COLUMNS="141"
ALSA_SINK=hw:1,0
LATENCY=500
PATH=gstreamer
OTHER_COMMAND_ON_LAUNCH=~/Laub-Woofer/junk.sh
#describe next client in the system
CLIENT=192.168.10.112, stereo
ACCESS=ssh pi@192.168.10.112
ALSA_SINK=hw:1,0
LATENCY=500
PATH=gstreamer
Recall that this is a system that uses gstreamer to stream audio over your LAN (a WiFi or wired connection) from a main "server" computer where your player resides to one or more "client" computers. Each system can be comprised of one or more clients. The input signal is assumed to be stereo audio. I make use of ALSA and its "loopback" feature to facilitate input on the "server" and output on the "client". Currently the system assumes that the input and stream formats do not change and are known in advance, meaning that bit-perfect playback is not possible except for audio that is already in the same format as the input and stream.

The meaning and use of each line is as follows:
  • AUDIO_SOURCE - this defines where to get the audio input. The string is input in gstreamer format exactly as it will be used in the gstreamer pipeline. Here I use the "dsnoop" ALSA device, which allows multiple simultaneous connections to a device.
  • AUDIO_SOURCE_CAPS - "caps" is short of capabilities and defines the sample rate, bit depth, etc. of the input.
  • PORT - the stream is sent to a port at the client's IP address. This defines the destination port for all clients in this system. Typically port 1234 is used, commonly used for UDP data.
  • STREAM_BITS and STREAM_RATE - this sets the bit depth and sample rate for the audio stream. Bit depths can be one of 16 or 24 bits. This is because RTP is used to send time-synchronized data packets and only these bit depths can be accommodated by RTP. The stream rate can be any integer rate but since there is no resampling in the client pipeline you need to choose a rate supported by your client's output device (e.g. DAC) if you want it to function.
  • VOLUME_TRIM_DB - I included this feature to allow the volume of an entire system to be adjusted, e.g. because for the same player volume setting it is too loud or too soft compared to other systems, etc. The current range is +6dB to -30dB in 1dB steps.
  • CLIENT - the CLIENT keyword begins the description of client related parameters. Prior to the first client keyword all the parameters related to the source computer. On the client line the IP address of the client is provided (assumed to be static) and, separated by a comma, the channel assignment for this client is provided. Channel can be one of stereo, left, right, or mono (stereo mixed down to one channel).
  • ACCESS - this string defines how to connect to the client. I am assuming an ssh connection. I have tested the system both with ssh using pre-generated shared public keys and the sshpass program. Sshpass is very simple to use but exposes the password because it must be stated in the access string in plain text. The access text also includes any environmental variables that need to be set in the ssh session. I have provided a few that helped my client application (see below) launch properly.
  • ALSA_SINK - since we know the input format and the audio stream format the client's gstreamer pipeline can mostly be autogenerated except for the output. Here the user is providing the ALSA device to which the decoded audio stream should be sent.
  • LATENCY - on each client the incoming RTP data must be buffered. The buffer size (in milliseconds) of the client can be independently set of other clients. This can be adjusted to trim out delay between clients to balance out their own inherent latencies. Here 500 msec is very generous because I am using a WiFi connection. This can be less than 100 msec over WiFi or perhaps 20msec or less if streaming over a wired LAN.
  • PATH - when logging into each client via ssh to launch the gstreamer pipeline, the user can cause the code to first cd to this directory. The path can be absolute or relative to the login directory. At this time, the directory must already exist.
  • OTHER_COMMAND_ON_LAUNCH - the user can supply a list of comma separated commands that will be run when the gstreamer pipeline is launched when streaming to the client starts up. For example, in my system I process the audio on each client using ecasound. Ecasound implements a crossover in the client. It's then directed by ecasound to DACs connected to the client. I use this field to specify a shell script that contains the ecasound command string.
  • OTHER_COMMAND_ON_TERMINATE - (not shown) like OTHER_COMMAND_ON_LAUNCH, this allows the user to specify commands that will be run on the client but these are run when the gstreamer pipeline is terminated when the system is "turned off" by the user from the server side. Commands that might be incorporated into OTHER_COMMAND_ON_LAUNCH and OTHER_COMMAND_ON_TERMINATE include setting the client's GPIO pins to a high or low state. These can be connected to relays that might turn on/off amplifiers, mute loudspeaker relays, illuminate an LED, or whatever you might find useful.

With the above command set, the user will be able to turn a system on or off remotely, that is through an easy to use text-based command interface that I will describe later. As part of turn on audio is streamed to a set of clients but the exact syntax of how this is done via gstreamer is hidden from the user. Because each system can be comprised of any number of clients, this is a very flexible setup in terms of what kind of systems can be controlled.

I will post more after some further development and coding.
 
Last edited:
Not sure what you mean by "managed on the client". In the example I gave:
Code:
ssh user@host ARG1="value1" ARG2="value2" /bin/bash << ENDSSH
  # commands to run on remote host
  # ARG1 and ARG2 are now environmental variables within the ssh session on the client
  lines of code that are run on the client go here...
ENDSSH
ARG1 and ARG2 become environmental variables on the client within and for the duration of that ssh session only.

As in your control script could just SSH over to the client, source a config file from some conventional location and then execute. Arguably this simplifies the control script as no longer has to worry about parsing parameters/vars.
 
As in your control script could just SSH over to the client, source a config file from some conventional location and then execute. Arguably this simplifies the control script as no longer has to worry about parsing parameters/vars.

Yeah, I tried to source .bashrc from the home directory on the client in a couple of different ways but could not get that to work. I don't think that more than a couple environmental parameters need to be passed so my way is lightweight enough not to be a burden. I'll stick with my current method for now, although I may revisit the issue. If you can point me to some concrete examples that would be helpful.
 
Yeah, I tried to source .bashrc from the home directory on the client in a couple of different ways but could not get that to work. I don't think that more than a couple environmental parameters need to be passed so my way is lightweight enough not to be a burden. I'll stick with my current method for now, although I may revisit the issue. If you can point me to some concrete examples that would be helpful.
ssh'ing into a box results in a non interactive shell so that won't typically load .bashrc, an answer like Why does an SSH remote command get fewer environment variables then when run manually? - Stack Overflow gives a decent summary of the situation and some of the answers give some typical solutions.

Personally I tend to try to avoid this sort of issue with 2 complementary approaches.

Firstly I tend to refactor my profiles (.bash_profile, .bashrc, /etc/profile etc, the precise files in play might vary with distro) so that both the .bashrc and the .bash_profile source a common file that define the important env vars.

For example, a simple setup is;

.bash_profile
Code:
. ~/.bash_common

.bashrc
Code:
. ~/.bash_common

.bash_common
Code:
# machine agnostic config
. ~/.core_envrc

# machine specific config if present
[ -e ~/.bash_machine ] && . ~/.bash_machine

Generally this is backed by a master config tree that is then sync'ed across a set of machines which means you could always source those locally before you execute a command on that particular remote machine.

Secondly you can write script that can load it's own env, for example you might include a snippet like in a script to ensure it always has that same set of environment variables. IME this is generally a problem with cron which may get an even more cutdown env

Code:
[[ -z "${SOME_VAR_NAME}" && -f "${HOME}/.bash_common" ]] && . "${HOME}/.bash_common"
 
ssh'ing into a box results in a non interactive shell so that won't typically load .bashrc, an answer like Why does an SSH remote command get fewer environment variables then when run manually? - Stack Overflow gives a decent summary of the situation and some of the answers give some typical solutions.

Personally I tend to try to avoid this sort of issue with 2 complementary approaches.

Firstly I tend to refactor my profiles (.bash_profile, .bashrc, /etc/profile etc, the precise files in play might vary with distro) so that both the .bashrc and the .bash_profile source a common file that define the important env vars.

For example, a simple setup is;

.bash_profile
Code:
. ~/.bash_common

.bashrc
Code:
. ~/.bash_common

.bash_common
Code:
# machine agnostic config
. ~/.core_envrc

# machine specific config if present
[ -e ~/.bash_machine ] && . ~/.bash_machine

Generally this is backed by a master config tree that is then sync'ed across a set of machines which means you could always source those locally before you execute a command on that particular remote machine.

Secondly you can write script that can load it's own env, for example you might include a snippet like in a script to ensure it always has that same set of environment variables. IME this is generally a problem with cron which may get an even more cutdown env

Code:
[[ -z "${SOME_VAR_NAME}" && -f "${HOME}/.bash_common" ]] && . "${HOME}/.bash_common"

I'm trying to develop a self-contained application. I do not want a user to have to go and edit various files on each client for it to work. An approach like your "refactored" files that refer to bashrc_common that must be implemented on or distributed to every client is not something I fine attractive and would just ask the user to do more setup work, requires them to customize their machines, etc. Also, I believe that your last bit of code is sourcing the bash_common file that you create - sourcing doesn't seem to work inside my here-document.

The way I do it now allows env vars for clients to be set as needed from the server side. This also localizes the user interaction to the configuration file on the server. The ssh session gets a subset of the usual env vars by default and the user only needs to supply a couple of additional ones as part of the ssh command string.

I appreciate your advice but I do not see how your approach is reducing barriers for the user or is in any way more powerful or capable than what I am doing.
 
Also, I believe that your last bit of code is sourcing the bash_common file that you create - sourcing doesn't seem to work inside my here-document.
you can source files within a here doc used with ssh, hard to say why it's not working for you though.

The way I do it now allows env vars for clients to be set as needed from the server side. This also localizes the user interaction to the configuration file on the server. The ssh session gets a subset of the usual env vars by default and the user only needs to supply a couple of additional ones as part of the ssh command string.

I appreciate your advice but I do not see how your approach is reducing barriers for the user or is in any way more powerful or capable than what I am doing.
IME approaches become useful as the number of vars goes up and/or when the values you want to pass around don't play nice with bash (e.g. when special characters are involved and you have to be able to escape them properly, the rules for which can get quite messy). If you don't have that sort of complexity to deal with then all you have, with this sort of solution, is more invasive/complex config to manage with no meaningful benefit (and hence not worth the effort).
 
you can source files within a here doc used with ssh, hard to say why it's not working for you though.


IME approaches become useful as the number of vars goes up and/or when the values you want to pass around don't play nice with bash (e.g. when special characters are involved and you have to be able to escape them properly, the rules for which can get quite messy). If you don't have that sort of complexity to deal with then all you have, with this sort of solution, is more invasive/complex config to manage with no meaningful benefit (and hence not worth the effort).

I agree with you. I don't expect that there will be anything more than plain text as the env var content. I also only expect the user to set a couple, since most are provided in the (non interactive) set of ssh env vars.

It would be nice to source the bashrc file. I will have to go back and take a look at this again. I believe I simple put
Code:
source ~/.bashrc
. It's possible that the home directory was not set, meaning the "~" would have failed. Not sure, since by that time I had figured out what the problem was (some env vars not set) and was just looking for some way to fix that.

If you think that sourcing bashrc in the here-document should work, I will just plug away at that and see if I can get it to work. I'm just hacking here, in no shape or form am I an accomplished bash programmer! In fact once this is working I would love for someone of that skill level to take a look at it for any massive security holes, or ways in which the code could be condensed or improved.

I pretty much have all the bits and pieces working now, and hopefully I will get it all put together in the next week or so. About the only thing I will improve at this point is the plain text terminal interface, where I would like to use some tput commands to improve the presentation, etc.
 
Update:
Success! I now have everything working including the user interface for system launch/termiante, etc. I plan to keep testing and improving the code over the next week or so. I would like to implement the text interface in a different way using tput. It's currently just outputting info as plain text which is probably fine, but could be made to look a little better.

At this point the concept is proven and I am very happy to see that my effort has come to fruition.
 
Here are some more usage details. The help screen shows some of the different ways that the streaming audio controller software can be used:

Code:
       *** GSASysCon: the Gsreamer Streaming Audio System Controller ***
      A bash-script-based streaming audio system controller for gstreamer

Help and Usage Information:

Usage:
  GSASysCon.sh                                  :run-once mode
  GSASysCon.sh (update_interval)                :continuous update mode
  GSASysCon.sh -a | -auto (system_number)       :automated mode
  GSASysCon.sh -h | --help                      :prints this help
  GSASysCon.sh -c | --copyright                 :prints (C) info
  GSASysCon.sh -v | --version                   :prints version info

Required parameters are shown within parenthesis.

Run-Once Mode:
In this mode the system status is displayed and the user can choose one action
to take. After performing the action the progam prints the new status and exits.
If no user input is received for some time the progam automatically exits.

Run-Continuous Mode:
In this mode the system status is continuously displayed after update_interval 
seconds have elapsed without user input. Like with run-once mode, the user can
choose an action to take. Following the action, the system status is again
displayed. The user must manually exit by typing x at the prompt.

Automated Mode:
This functions like run-once mode. Rather than getting the desired action from
the user, the action is specified on the command line. Actions can only consist
of turning a system on or off, so the only permissible input is a system number.

The progam lists the systems that have been registered, and their status. The
status is either 'OFF' or the elapsed up time of the system. Toggling a system
on and off is performed by entering the system number. The system number is 
automatically generated by arranging the systems in alphabetical order.

The status screen shows a list of the systems that are registered within GSASysCon. The output (for just one registered system) is shown below:

Code:
   #       STATUS        SYSTEM NAME
----------------------------------------------------------
   1   2hours,14mins     main system
----------------------------------------------------------
Enter a system number to toggle it ON/OFF, x to exit

I can control my audio system from a variety of devices. I use a 10" android tablet to run an mpd client (MPD Droid IIRC) and then I use an ssh client to access the server and run GSASysCon. Turning a system ON or OFF is as easy as typing its number into the GSASysCon interface. I can have any number of mpd and GSASysCon interfaces running on different devices, so that I can control any audio system in my home from which ever one is closest to where I happen to be at the time.

Because GSASysCon has an automatic mode it could be run as a cron job to turn a system on at a preset time, for instance as an alarm clock. Likewise, a cron job could shutdown all the systems at some time, e.g. after you fall asleep. It can be called from script, so automated audio on/off switching can be wrapped into other applications without the need for user input.
 
I'm following up about sourcing .bashrc or .profile so that environmental variables in them are set for the ssh session.

I did some more experimenting and was able to successfully source .profile, which calls bashrc. The problem was that the TERM variable still remains unset - this will generate an error with my client applicatino. Also the variable COLUMNS is not set to anything, which is also important for that application to function correctly.

For this reason I will just keep the current arrangement of having the user specify two or three crucial environmental variables as part of the ssh connection string. This works very reliably and I do not need to make any assumptions about what values the user might want to assign.
 
I have begun to implement the feature "run local script on client". This works in the following way:

Instead of storing a shell script containing programs/apps/commands to run after the gstreamer pipeline has been launched on the client, the user can store the same script on the server. I feel that this helps to centralize all the info for each system in one place (in the system's directory on the server).

Under my current setup I use this shell script to launch ecasound after the pipeline is up and running. Ecasound implements a DSP crossover in the client and sends the audio to the speakers via the client's DACs. I have a separate client (computer) in each loudspeaker and subwoofer module. This means that the right and left speakers use the exact same crossover filters, so the exact same ecasound script file. The gstreamer pipeline handles separating the stereo input into left and right channels and streaming the correct (single channel) audio to each speaker. If I keep the script files on each client and I want to make a change I need to log into each client and update its ecasound script file. On the other hand if I have this on the server I only need ONE file because both left and right speakers use the same crossover. This simplifies tweaking the crossover or other changes that might want to make. I can do this from my listing chair, because I am already connecting the the server to use the control interface that is the subject of this thread. It's a simple matter to open a parallel window, edit the ecasound script file, and then stop and start the stream to implement the changes.

By pairing up the gstreamer-based streaming audio application with ecasound as a DSP crossover platform the DIYer can build an awesome all-digital wireless loudspeaker system.
 
I want to emphasize that "client computers" only need to be something like a Raspberry Pi model 2 or 3 ($35) or a Beagle Bone Black ($50?).

For the Raspberry Pi, using gstreamer to stream PCM (uncompressed) digital audio to the client gets around one of the problems of that platform, namely the lack of a good quality audio input (A to D converter) that is suitable for high quality stereo audio. The hardware that is required for setting up an 802.11g 2.4G network is very inexpensive these days. As an example, I have a USB dongle ($15) on each client getting WiFi from an 802.11g/n WiFi router ($50) and this allows me to stream 24/96 to multiple clients. Since the gstreamer app can split up streams into mono, L, R this reduces bandwidth requirements somewhat and I probably could do a higher rate (although personally I don't see a need for that and I only have 24/96 capable DACs). The Pi includes an ethernet (wired) connection for those who do not wish to use WiFi but would like to use the Pi's DSP processing capabilities (via ecasound).
 
Last edited:
I thought I would write down some more info about the streaming audio system and what you would use to get it working.

The GSASysCon software:
GSASysCon stands for the "Gstreamer Streaming Audio System Controller". The code that runs everything is a bash shell script, so this requires Linux, however, there is a chance that it will run under Cygwin (this is not tested). Bash commands are used to launch and terminate (e.g. kill) gstreamer pipelines and other things, so everything should be run under some Linux variant. Gstreamer 1.x is used to stream the audio (I use version 1.6), and this is available for multiple OSes. For more info, see:
https://en.wikipedia.org/wiki/GStreamer
I have been testing the code under Ubuntu 15.10 on the audio server and the latest Raspbian distros on each client.

Hardware:
An audio "server" computer supplies audio. One or more "client" computers receive the audio stream and must be able to render it to analog through DACs or otherwise provide some way to drive speakers (USB amplifier?). Streaming can be done over any network, wireless or wired.

Other software:
I strongly suggest using ALSA, and an ALSA loopback (the snd-aloop module). Under this scenario, a source supplies audio to the server's audio loopback (input) and the loopback (output) is connected to the gstreamer pipeline that streams audio from the server. On the client(s), gstreamer receives the audio stream and sends the audio to the client's loopback (input) if another program (like ecasound) needs to further process it or it can send the audio directly to other ALSA hardware like a DAC.

On the server, a "player" is used to supply audio that will be streamed by the GSASysCon application. Unless you want to be required to be sitting at the server to start stop or change tracks, volume , etc. then you want some kind of player software that works on mobile devices. For example, MPD runs on my server and I use a Android tablet top run an "MPD client", an interface for controlling MPD. I can have this with me at my listening location or wherever I am in my home. I can also control GSASysCon from there by logging into the audio server. I use ssh for that purpose, but I recently found a cool app that runs an ssh interface inside a browser, which makes things very convenient.

A Word about Audio Quality and Bit Perfect Playback:
Audio is streamed as PCM at either 16 or 24 bit resolution. The sample rate is unlimited by the software but likely limited in the real world by the maximum SR of your rendering device (the DAC) or the bandwidth of your LAN connection. Currently the input format (bit depth and sample rate) must be known, and typically everything is resampled to some fixed format of the user's choosing. Therefore, bit perfect playback is not possible for all formats. This is because audio is streamed under the RTP protocol and this is not a dynamic format and can not adapt to changing audio formats. As a result you need to inform the pipeline what format it will be carrying when you launch it, and the format remains fixed for the life of the connection.

Uses and Capabilities:
No need to run wires all over the place if you use WiFi. Various components of the audio system can be independently synchronized (like subs and main speakers, or whatever). There is no limitation (other than those imposed by your hardware) on how complex the audio system can be. One interesting application would be for multiple distributed subwoofers. The GSASysCon application allows the user to launch "other" applications (as shell scripts) when the streaming audio pipeline is either launched or terminated. I plan to use this to turn on some DSP processing at my clients and to switch on/off speaker relays and power amps at the loudspeaker(s) via relays driven by the GPIO pins of my R-Pi clients. I recently wrote some LADSPA plugins for this purpose, but you could do it directly using the pigpio library or other way to toggle the gpio pins from the command line (there are several).

Because the system can run over a wireless LAN, you can move systems around to a new location at will. This could include moving a system outdoors (like a DIY boom box) and then streaming your music collection to it over WiFi. I mentioned before that there is an automated mode that could be used to turn on a system like an alarm clock and then launch a player application to play some music or perhaps turn off systems at some time (when you have fallen asleep). Use your imagination.

One could have multiple versions of each system with different parameters (bit depth or sample rate, volume trim, using a subwoofer or not, etc.) and switch back and forth or A/B compare.


Final Thoughts:
At this point the system seems to be fully working. It would be good if I could work with one or two people who would be interested in beta testing it, in case I have not found a bug or something about the system is difficult to grasp, set up, etc. Apart with some familiarity with Linux there are no special requirements or expertise needed. I would like to see if a user can go about the setup process with some instruction to make sure the barriers to get up and running are low.

If you would like to be part of the beta testing, please drop me a PM.
 
Hmmh. Lots of effort. Not sure if I really understood what you're up to.

I'm running a Logitechmediaserver and several squeezelite clients on PIs, PCs and streaming clients on phones and tablets.

All the features and control apps are awesome. The streaming environment is rock solid.
MPD based systems can't compete.

On active speakers I used to run squeezelite piped into ecasound which is running LADSPA.

For remote power-on/power-off I use wireless switches.


By using squeezelite I do have numerous options to tweak the streamer. E.g. full file buffering and a hundred more tweaks for better sound.

Running LMS gives me an easy option to do 2-channel DSP work on powerful server on a per client basis.

Anything else I'm looking for??

Yep. Active wireless network speakers, which can be kept in sync.

Beside that, I'm wondering what your solution could add to my setup !?!?
 
Hmmh. Lots of effort. Not sure if I really understood what you're up to.

I'm running a Logitechmediaserver and several squeezelite clients on PIs, PCs and streaming clients on phones and tablets.

All the features and control apps are awesome. The streaming environment is rock solid.
MPD based systems can't compete.

On active speakers I used to run squeezelite piped into ecasound which is running LADSPA.

For remote power-on/power-off I use wireless switches.


By using squeezelite I do have numerous options to tweak the streamer. E.g. full file buffering and a hundred more tweaks for better sound.

Running LMS gives me an easy option to do 2-channel DSP work on powerful server on a per client basis.

Anything else I'm looking for??

Yep. Active wireless network speakers, which can be kept in sync.

Beside that, I'm wondering what your solution could add to my setup !?!?

Well, thanks for all of your... um... "encouragement".

Yes, the gstreamer based system can provide well-synchronized systems, and even synchronization within a system. If you read the thread, you should see that I use multiple clients within each system (left speaker, right speaker, subwoofer for example: 3 clients). These are all kept in tight sync. That's just not possible with other streaming systems from my experience.

Honestly, streaming stereo audio to a single client is trivial, and there are a myriad of "solutions" for that (Airplay, DLNA, squeezelite, VLC, etc). But add another client right next to it and what do you have? An out of sync mess. Gstreamer has excellent clock management via RTP streaming, and that is one of the reasons why I have been using it, and developed this application around it.
 
Well, thanks for all of your... um... "encouragement".

Well. The weaknesses of a solution pop up if you start looking under the hood.

As you said. Streaming a file is not an issue.

Making it right, by meeting more than one criteria, is the challenge.

well-synchronized systems

How well??

I mean, keeping two PIs and attached DACs properly in sync looks quite challenging to me.
Most solutions I've seen still need a left/right speaker sync on a DAC/clock level.


Honestly, streaming stereo audio to a single client is trivial, and there are a myriad of "solutions" for that (Airplay, DLNA, squeezelite, VLC, etc).

Hmmh. I'm in streaming "business" for quite some time. The "only" serious solution I see is
the squeeze environment.
E.g. I havn't seen any DLNA solution that would meet my criterias. E.g. Usability is typically way below the squeeze environment.
 
I mean, keeping two PIs and attached DACs properly in sync looks quite challenging to me.
Most solutions I've seen still need a left/right speaker sync on a DAC/clock level.

That is exactly right - you need synchronicity on the clock level. Here is how I do this when there are multiple clients involved:

1. The streams must be able to be rendered in sync. I use the RTP protocol for this purpose. RTP includes timestamps that are generated at the server side and these are used at the client side to control the rate that the stream is rendered to the local (internal) audio system, which in my case is ALSA.

2. I use DAC that get their clock from the USB bus via USB adaptive mode. The DAC gets its clock from the bus, and the bus timing is derived from the host system clock.

3. I construct the client gstreamer pipeline to spit out a rate-controlled stream. The rate is derived from the system clock.

4. I use NTP to synchronize the clocks of the server and all clients in my system. To provide low latency I added an NTP server to my network using a Raspberry Pi Zero. This is set up with the standard polling of some regional NTP timeservers. The goal is NOT to get accurate time/date on each client, but rather to get their clocks running at the same RATE. The actual time of day is irrelevant. By setting my NTP server to poll internet timeservers on a long time scale, and setting all local computers to poll my server on a very short timescale, everything can be kept in sync to within a couple hundred microseconds or better. For example, two clients in one of my systems are current showing -0.038 msec and -0.044 msec from true time. That's a difference of only 0.006 msec or 6 microseconds. That is really pretty darn good.

When all system clocks are running at the same rate, their USB busses are also running at the same rate, and the USB adaptive mode DAC clocks are all running at the same rate.

Using the "right" hardware is important. Using high end higher priced DACs doesn't help or may hurt. Many DACs these days are asynchronous in the zealous school of thought that any jitter must be absolutely horrible. But these will not work in my system because they have their own clocks, which eventually will go out of sync. Also, in reality I do not need stratospheric sample rates and a 192k or 384k DAC seems to be overkill to me in general.

I use ES9023 based DACs, sometimes as an integrated USB/DAC and sometimes with separate USB->I2S and I2S->DAC boards. These are low cost and perform well. I have a bunch of 16/48k DACs and have a couple 24/96k DACs as well, and all are USB adaptive mode types and are quiet inexpensive. I can stream these formats over WiFi without a problem with excellent synchrony.
 
I made a couple of improvements over the weekend and will continue to clean up the code and test for any last bugs or unintended behavior.

My main focus this week is to make some progress on the documentation and implementation guide. I'm about 20% done with it at this point.

Have been testing it over the weekend. I set up three different systems and this helped me to make a couple of small improvements that allow the code to autodetect new system definitions on launch. Otherwise everything seems to be working great and hopefully version 1.0 can be released in the next week or so.
 
I made a couple of improvements over the weekend and will continue to clean up the code and test for any last bugs or unintended behavior.

My main focus this week is to make some progress on the documentation and implementation guide. I'm about 20% done with it at this point.

Have been testing it over the weekend. I set up three different systems and this helped me to make a couple of small improvements that allow the code to autodetect new system definitions on launch. Otherwise everything seems to be working great and hopefully version 1.0 can be released in the next week or so.


Awesome news Charlie, can't wait to try it out soon.