BruteFIR setup for windows?

[phofman]

You can chain multiple sox effects in single sox instance SoX

Sox can read flac files, why the flac binary?

 

[scott wurcer]

Too bad it does not process files in parallel, but it write to output all at once at the end, so is unsuitable for real time (i.e applying FIR to streamed pcm from Qobuz or Tidal).

With Large WAV files (600 MB) it takes few seconds on my i5 with windows 10.

Any Idea?

I doubt a 600MB file is being processed in series twice by the fir's to completion before you get output. As others have suggested there are probably pipe related things you don't have control of. If you run the fir on a 600MB stereo file and pipe it to sox.play what is the latency?

The fir filter is implemented by the overlap and add FFT technique and will have some latency but I would think that would be < sec.

 

[marcoc1712]

You can chain multiple sox effects in single sox instance SoX

Sox can read flac files, why the flac binary?

was just an example, FLAC is mainly used to cut file with a start and an endpoint depending on cue files, SOX could do the same, but with much more latency than FLAC (or FFMPEG).

 

[scott wurcer]

You can chain multiple sox effects in single sox instance SoX

That was an early suggestion but the fir is only implemented as a stereo effect (AFAIK). I could not think of a math way using the properties of convolution to do it in an effects chain, I might be missing something.

 

[marcoc1712]

I doubt a 600MB file is being processed in series twice by the fir's to completion before you get output. As others have suggested there are probably pipe related things you don't have control of. If you run the fir on a 600MB stereo file and pipe it to sox.play what is the latency?

The fir filter is implemented by the overlap and add FFT technique and will have some latency but I would think that would be < sec.

,
Not sure to well understand what you are asking me to do, but:

sox -M -t sox "|sox.exe Adagio.wav -p remix 1,1 fir shelvingHigh.txt" "|sox.exe Adagio.wav -p remix 2,1 fir shelvingLow.txt" -b 24 Adagio_out.wav

takes 21 seconds, being Adagio.wav 680 MB and Adagio_out.wav is empty until the very last second, then it fills up all at once.

then, playing Adagio_out.wav is immediate.

If I run sox -M -t sox "|sox.exe Adagio.wav -p remix 1,1 fir shelvingHigh.txt" "|sox.exe Adagio.wav -p remix 2,1 fir shelvingLow.txt" - | player.exe

where player.exe is squeezebox server module that get the input from his stdin, it starts playing (not really, should need some format adjustment here, but it starts receiving input, ) after 21 secs.

 

[DonVK]

The pipe operator is "|" and the chain effect operator is ":" .

I originally tried using the effect chain as it made more sense to me, but could not get the syntax to get it to work.

@phofman : could you provide an example of channel splitting, fir effect chain with unique fir to each channel, then merging back ?

 

[scott wurcer]

,
Not sure to well understand what you are asking me to do, but:

I see that's not good it looks like SoX can't set this up efficiently without help.

You could recompile SoX from source with a stereo fir filter (like foo_convolve). The fir would be a .txt file with two entries per line. I doubt this solution appeals to you.

IIRC SoX supports Ladspa plug-ins, maybe there is one that can do this.

 

[marcoc1712]

That was an early suggestion but the fir is only implemented as a stereo effect (AFAIK). I could not think of a math way using the properties of convolution to do it in an effects chain, I might be missing something.

FIR is implemented as MONO only, applied to any and every channel in the input file.

 

[phofman]

My fault, I did not realize you needed a different effect to each channel. I do not think sox can do that without the the piped multiple input feature.

IMO the reported behavior on windows is direct result of the way pipes work in that OS (e.g. Information Technology: An Introduction for Today’s Digital World - Richard Fox - Knihy Google ) - synchronous first all in, then all out. I have no Windows available to test now.

The piped input is a nice way on systems with standard pipes since all the processing runs in parallel. Sox itself is single-threaded - a longer serial chain would take longer.

 

[scott wurcer]

FIR is implemented as MONO only, applied to any and every channel in the input file.

Sorry that's what I meant both channels at once with same fir.

 

[marcoc1712]

My fault, I did not realize you needed a different effect to each channel. I do not think sox can do that without the the piped multiple input feature.

IMO the reported behavior on windows is direct result of the way pipes work in that OS (e.g. Information Technology: An Introduction for Today’s Digital World - Richard Fox - Knihy Google ) - synchronous first all in, then all out. I have no Windows available to test now.

The piped input is a nice way on systems with standard pipes since all the processing runs in parallel. Sox itself is single-threaded - a longer serial chain would take longer.

I was wrong, the problem is elsewhere, I should investigate where, but the command:

sox -M -t sox "|sox.exe Adagio.wav -p remix 1,1 fir shelvingHigh.txt" "|sox.exe Adagio.wav -p remix 2,1 fir shelvingLow.txt" -

produces an (almost) immediate output to stdout, so seems it works in parallel, even in windows.

Now I face another problem: how to feed the two pipes from the common stdin instead than from a file (use case is a stream form Qobuz or Tidal)

I tried this:

sox -M -t sox "|sox.exe - -p remix 1,1 fir shelvingHigh.txt" "|sox.exe - -p remix 2,1 fir shelvingLow.txt" -t wav -

that seems to work, but the result is a weird stereo file in output, completely out of sync


F:\SVILUPPO\06 - sox>sox --info stereo_in.wav

Input File : 'stereo_in.wav'
Channels : 2
Sample Rate : 44100
Precision : 16-bit
Duration : 00:00:20.00 = 882000 samples = 1500 CDDA sectors
File Size : 3.53M
Bit Rate : 1.41M
Sample Encoding: 16-bit Signed Integer PCM


F:\SVILUPPO\06 - sox>sox --info out.wav

Input File : 'out.wav'
Channels : 2
Sample Rate : 44100
Precision : 32-bit
Duration : 00:00:10.40 = 458752 samples = 780.19 CDDA sectors
File Size : 3.67M
Bit Rate : 2.82M
Sample Encoding: 32-bit Signed Integer PCM​

here the real command and the log:

F:\SVILUPPO\06 - sox>"F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\flac.exe" -dcs --force-raw-format --endian=little --sign=signed -- "F:\SVILUPPO\01 - SqueezeboxServer Plugins\musica campione\20_Sec\wav_16_044100.flac" | "F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe" -V6 -M -t sox "|"F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe" -V6 -t raw -c 2 -b 16 -r 44100 -e signed-integer -L - -p remix 1,1 fir shelvingHigh.txt" "|"F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe" -V6 -t raw -c 2 -b 16 -r 44100 -e signed-integer -L - -p remix 2,1 fir shelvingLow.txt" -t wav -c 2 -b 32 -r 44100 out.wav
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe: SoX v14.4.2
time: Feb 25 2017 23:16:31
compiler: gcc 4.9.1
arch: 1248 48 88 L
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe: SoX v14.4.2
time: Feb 25 2017 23:16:31
compiler: gcc 4.9.1
arch: 1248 48 88 L

Input File : '-' (raw)
Channels : 2
Sample Rate : 44100
Precision : 16-bit
Sample Encoding: 16-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no


Output File : '-' (sox)
Channels : 1
Sample Rate : 44100
Precision : 32-bit
Sample Encoding: 32-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'

F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG effects: sox_add_effect: extending effects table, new size = 8
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG remix: 0:
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG remix: 1 0.5
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG remix: 0 0.5
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO fir: 16384 coefficients
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: input 44100Hz 2 channels (multi) 16 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: remix 44100Hz 1 channels (multi) 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: fir 44100Hz 1 channels 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: output 44100Hz 1 channels (multi) 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG sox: start-up time = 0.022938
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO formats: detected file format type `sox'
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe: SoX v14.4.2
time: Feb 25 2017 23:16:31
compiler: gcc 4.9.1
arch: 1248 48 88 L

Input File : '-' (raw)
Channels : 2
Sample Rate : 44100
Precision : 16-bit
Sample Encoding: 16-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no


Output File : '-' (sox)
Channels : 1
Sample Rate : 44100
Precision : 32-bit
Sample Encoding: 32-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'

F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG effects: sox_add_effect: extending effects table, new size = 8
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG remix: 0:
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG remix: 0 0.5
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG remix: 0 0.5
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO fir: 16384 coefficients
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: input 44100Hz 2 channels (multi) 16 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: remix 44100Hz 1 channels (multi) 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: fir 44100Hz 1 channels 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: output 44100Hz 1 channels (multi) 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG sox: start-up time = 0.021942

Input File : '|F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe -V6 -t raw -c 2 -b 16 -r 44100 -e signed-integer -L - -p remix 1,1 fir shelvingHigh.txt' (sox)
Channels : 1
Sample Rate : 44100
Precision : 32-bit
Sample Encoding: 32-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'


Input File : '|F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe -V6 -t raw -c 2 -b 16 -r 44100 -e signed-integer -L - -p remix 2,1 fir shelvingLow.txt' (sox)
Channels : 1
Sample Rate : 44100
Precision : 32-bit
Sample Encoding: 32-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'

F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: Overwriting `out.wav'
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG wav: Writing Wave file: Microsoft PCM format, 2 channels, 44100 samp/sec
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG wav: 352800 byte/sec, 8 block align, 32 bits/samp

Output File : 'out.wav'
Channels : 2
Sample Rate : 44100
Precision : 32-bit
Sample Encoding: 32-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'

F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG effects: sox_add_effect: extending effects table, new size = 8
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: input 44100Hz 2 channels (multi) 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe INFO sox: effects chain: output 44100Hz 2 channels (multi) 32 bits unknown length
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG sox: start-up time = 0.095743
F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe DBUG wav: Finished writing Wave file, 3670016 data bytes 458752 samples​

I'm stuck.

 

[marcoc1712]

Just in case someone would like to have a look to the result (out.wav) Here the link to the zip containing it, the source and filter files I used in the command:


"F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\flac.exe" -dcs --force-raw-format --endian=little --sign=signed -- "F:\SVILUPPO\01 - SqueezeboxServer Plugins\musica campione\20_Sec\wav_16_044100.flac" | "F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe" -V6 -M -t sox "|"F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe" -V6 -t raw -c 2 -b 16 -r 44100 -e signed-integer -L - -p remix 1,1 fir shelvingHigh.txt" "|"F:\Sviluppo\slimserver\Bin\MSWin32-x86-multi-thread\sox.exe" -V6 -t raw -c 2 -b 16 -r 44100 -e signed-integer -L - -p remix 2,1 fir shelvingLow.txt" -t wav -c 2 -b 32 -r 44100 out.wav


sox.zip - Google Drive

Many thanks.

 

[scott wurcer]

I'm sorry your latest step is beyond my experience. I only play files from my hard drive. I have nothing more to suggest.

 

[phofman]

I do not use windows. In linux all this would be quite simple. I would suggest to consider the new WSL in windows Install Windows Subsystem for Linux (WSL) on on Windows 10 | Microsoft Docs and run the chain in "linux" terminal. There you have the tee command, real pipes, named pipes (mkfifo), configurable buffers etc. Your computer will become your servant, not your master.

 

[marcoc1712]

I do not use windows. In linux all this would be quite simple. I would suggest to consider the new WSL in windows Install Windows Subsystem for Linux (WSL) on on Windows 10 | Microsoft Docs and run the chain in "linux" terminal. There you have the tee command, real pipes, named pipes (mkfifo), configurable buffers etc. Your computer will become your servant, not your master.

Will you use a different command in Linux? If so witch one or different in what?

 

[DonVK]

It occurred to me, that the issue you are having is exactly the same as those trying to use SOX to make an XO. The channel(s) are split, processed, then remixed. Hopefully they are still in sync at the end. The XO just uses a different sox effect and sometimes includes the fir effect.

This thread has a nice explanation of the issues and some script examples : SoX - implementing loudspeaker crossovers using Sox + LADSPA plugins .

 

[marcoc1712]

It occurred to me, that the issue you are having is exactly the same as those trying to use SOX to make an XO. The channel(s) are split, processed, then remixed. Hopefully they are still in sync at the end. The XO just uses a different sox effect and sometimes includes the fir effect.

This thread has a nice explanation of the issues and some script examples : SoX - implementing loudspeaker crossovers using Sox + LADSPA plugins .

THX I'll have a look to that page. I'm giving a try to Brutrfir at moment and it works just fine in Linux.

 

[DonVK]

multi channel realtime FIR in SoX - not

Sox can do multichannel FIR if the I/O is a file. The problem in real time is piping data into multiple processing paths and actually have it remain in sync for output mixing. I used a 2 way stereo cross over to generate 4 channels (LL, LH, RL, RH) to check if sync was possible.

There is a wonderful discussion on SoX for crossovers and pipes at SoX - implementing loudspeaker crossovers using Sox + LADSPA plugins and I had a chance to try some of those ideas. It's only to aid my understanding of sox as it seemed like a solution was so close. Spoiler alert.

The input pipe is progressively split into multiple pipes using the TEE command. The link above explains the issues with pipes and blocking and requires this structure. A "pipe-zilla" tree is created that could be fed from a single file or a soundcard (dsnoop) or another process and each process chain gets its own fifo.

mkfifo llo.sox 
mkfifo rlo.sox 
mkfifo lhi.sox 
mkfifo rhi.sox 
mkfifo split1.fifo 
mkfifo split2.fifo 
mkfifo split3.fifo 


# Notice a pattern?  You could add more FIFOs into this chain :) 
(tee <split1.fifo split2.fifo | cat >lhi.sox) & 
(tee <split2.fifo split3.fifo | cat >rlo.sox) & 

# "sox -M" actually reads from this, first 
(cat < split3.fifo > rhi.sox) & 

# Onesubscriber feeds multiple pipes: 
(sox $INPUT -p | tee split1.fifo | cat > llo.sox) &

I tried 3 different processing chains to create the output and compare the outputs.

The first (8a) uses R. Taylor's LADSPA plugins and it works well.

 sox -S -V3 -M -t sox \
	"|sox llo.sox -p remix 1,1 ladspa RTlr4hipass $XOfc  " \
	"|sox lhi.sox -p remix 1,1 ladspa RTlr4lowpass $XOfc " \
	"|sox rlo.sox -p remix 2,1 ladspa RTlr4hipass $XOfc  " \
	"|sox rhi.sox -p remix 2,1 ladspa RTlr4lowpass $XOfc " \
	-t wav stereoout8a.wav \

The second (8b) uses the sox built in filters and it works well.

 sox --multi-threaded -S -V3 -M -t sox \
	"|sox llo.sox -p remix 1v1 lowpass  -$XOorder $XOfc $XOq"q" lowpass  -$XOorder $XOfc $XOq"q" " \
	"|sox lhi.sox -p remix 1v1 highpass -$XOorder $XOfc $XOq"q" highpass -$XOorder $XOfc $XOq"q" " \
	"|sox rlo.sox -p remix 2v1 lowpass  -$XOorder $XOfc $XOq"q" lowpass  -$XOorder $XOfc $XOq"q" " \
	"|sox rhi.sox -p remix 2v1 highpass -$XOorder $XOfc $XOq"q" highpass -$XOorder $XOfc $XOq"q" " \
	 -t wav stereoout8b.wav

The third (8c) uses the FIR effect in sox and 16K tap FIR filters. It does not work and appears to be blocked. It seems the fir effect requires a file and does not work with pipes (error - can't open pipe error on ctrl-C to interrupt blockage). [Sigh] I think this is a dead end that requires code changes to the SoX fir effect. I'm going back to Ecasound and Brutefir.

 sox -S -V3 -M -t sox \
	"|sox llo.sox -p remix 1,1 fir $LowPassFir " \
	"|sox lhi.sox -p remix 1,1 fir $HighPassFir " \
	"|sox rlo.sox -p remix 2,1 fir $LowPassFir " \
	"|sox rhi.sox -p remix 2,1 fir $HighPassFir " \
	-t wav stereoout8c.wav \

The scripts used are attached and the FIR filters and music file are in a previous post BruteFIR setup for windows?

 

[scott wurcer]

I'm going back to Ecasound and Brutefir.
[/URL]

Wow a lot of work, thanks. Still I have the question, why not a LADSPA plugin that does multi channel effects internally without any splitting of streams to mono and recombining? Understanding the overlap and add FFT fir is not rocket science.

 

[DonVK]

I thought that different processing for each channel (eq or fir) was the goal here. I was assuming that the DRC or EQ solution was unique to each channel hence the split, process and merge. Otherwise the same SoX effect or LADSPA plugin would be applied uniformly to an entire set of channels by default.

I did look for a LADSPA convolution engine that permits an external FIR file and could not find one. There are some impulse response based plugins that have internal presets for amp or room modelling but you can't specify your own file. Mostly I was just interested in what sox could do.

I still find Linux audio confusing because there are many places and ways to do the same function and often the documentation is lacking.