Would baffle dimension change of 2.5mm in all 4 sides change sound in a significant enough way?

It is possible that adding 2.5mm of veneer to the walls might slightly increase their rigidity and also change their resonance characteristics. But I strongly doubt you'd hear much difference unless the original walls were pretty flexible or light (e.g. foam-board).

The tiny dimensional change would (IMO) make no audible difference in a typical speaker cabinet.