low cost TDM ADC/DAC

Are saying you need a total of 32 ADC channels and 32 DAC channels (or maybe 16 of each) and you want to interface over I2S bus to a DSP chip?
Yes
No USB or other interface?
The idea of the project is build an very modular usb audio interface. The board is the Infinitas from freedsp.cc. this board is only able to get tdm over i2s.The main board DSP is an USB interface. So USB doesn't make very sense. But if easy and cheap to convert to i2s can be another interface.

Are these 16-bit data converters or 24-bit?
I prefer 25-bit but 17 is nice too.
 
What is commonly referred to as 'I2S bus' is different from I2S protocol. Presumably the OP means that TDM protocol is being used on what might be loosely called, "I2S bus."

If there is a preferred name for the bus it would be fine to use that.