What do you mean by "quite a lot of noise"?
Ok, so a rather vague description but the problem is usually the nature rather than amplitude. Most smps are very quiet at audio frequencies but produce rf noise over a wide bandwith (presumably harmonics of the switching frequency) - visible on a good scope. The other issue is radiation from components within the power supply and cabling.
Your point about opamp circuits is valid in that the noise produced by smps is insignificant in many applications. Most mass market audio equipment uses this sort of technology for reasons of cost, efficiency and reliability versus ultimate sound quality.
Anyone building a 300b amp is looking for the sort of sound quality that mass market stuff can't deliver (£1000+ spend on components alone) and at this level small design details have a dramatic effect on the final results.