Any problem Using steel to mount driver by magnet?

Hey all. I’m wanting to not use m much wood in my next project (had to build too many shelves... need a break from wood) and was thinking of mainly using black steel plumbing pipe for an OB project.

Anyhow I was going to mount the drivers by the magnet by gluing them to pipe flanges but thought having a huge chunk of ferrous material right there might not be so great for the driver. You guys have any thoughts on this? Will it make a difference at all to driver performance? I’ll measure response in the process of doing the active crossover, so any bulk shifts in response are fine, just don’t want to introduce things I can’t correct for.

Thoughts?

Thanks!!
 
I only have experience with AlNiCo mounted to galvanized and stainless steel, so I guess it's a matter of whether there's any magnetic attraction. If none, then can you clamp it with heat conducting paste to use it as a heat sink like I've done?
 
IMG_20201116_192641.jpg
Something like this ? - it's a car dual concentric, with the tweeter removed so there's a hole through the centre pole piece that the threaded bar goes through.
 
I used something similar to this on some small 2 ways, but it was threaded aluminum rod that went through the pole piece into a phase plug with internal thread. I did notice a small measureable reduction in efficiency and decreased Qes/Qts, maybe due to the lack of dust cap and more electrical dampening from the phase plug being close to the VC gap.
 
In my previous image, the threaded bar is brass ( I can't remember if this was intention or I just had some lying around ) and there might be a non conductive spacer between the pole piece and " bullet ", but I cant remember. What about using a lump of concrete as a mass ?
I was going to mount the whole thing on tension springs, with a foam gasket between the driver and front baffle.