Depends how tightly you define damage. You could use superglue or clear nail varnish, and then blot with acetone (nail polish remover) to remove the glue. I imagine this would leave residue and at least stain the paper. I'd practice on test scraps first, but I doubt it'd do any lasting harm to the sound 'quality' of the whizzer.
I have some old (butchered) Wharfedales I could try a whizzer rebuild on - with low expectations. If well-equipped companies make $800 drivers with FR plots that look like a mountain range, I doubt I could do better with glue and tweezers.