Issues with Pi 4, DACs and VLC

Did you try the plug plugin with vlc and check in hw_params if the hw plug converts to mmap mode?

Apparently vlc has problems with stream timestamps reported by alsa, not with the actual streaming. Maybe in rw mode the I2S driver provides some wrong timestamps. Therefore I would try vlc -> some plugin allowing to go from RW to mmap -> i2s hw.
 
Code:
$ aplay -v -D test --period-size=1764 --buffer-size=131072 -r 44100 -c 2 -f S32_LE Music/sample-32-3.wav
Playing WAVE 'Music/sample-32-3.wav' : Signed 32 bit Little Endian, Rate 44100 Hz, Stereo
Plug PCM: Hardware PCM card 1 'E30' device 0 subdevice 0
Its setup is:
  stream       : PLAYBACK
  access       : RW_INTERLEAVED
  format       : S32_LE
  subformat    : STD
  channels     : 2
  rate         : 44100
  exact rate   : 44100 (44100/1)
  msbits       : 32
  buffer_size  : 131072
  period_size  : 1764
  period_time  : 40000
  tstamp_mode  : NONE
  tstamp_type  : MONOTONIC
  period_step  : 1
  avail_min    : 1764
  period_event : 0
  start_threshold  : 131072
  stop_threshold   : 131072
  silence_threshold: 0
  silence_size : 0
  boundary     : 1073741824
  appl_ptr     : 0
  hw_ptr       : 0


Code:
$ more /proc/asound/card1/pcm0p/sub0/hw_params
access: RW_INTERLEAVED
format: S32_LE
subformat: STD
channels: 2
rate: 44100 (44100/1)
period_size: 1764
buffer_size: 131072