A good speaker driver will reproduce equal amplitude over the frequency range that you want to use it for. The flatter the response the more accurate. That is one of the most important features of any driver. Some drivers will ring or slew at the frequencies where the amplitude response rolls off, or has a big amplitude peak, so it's generally wise to choose crossover frequencies that roll off the driver electronically before you get near frequencies that the driver can't do well.
Another area to know about is off axis response. An 8 inch driver starts to get significantly directional around 1kHZ. If you cross an 8 inch woofer over to a 1 inch dome tweeter at 3kHZ for example, the off axis response will have an abrupt jump in the amplitude versus frequency response at the crossover frequency. Off axis at 3kHZ the 8 inch woofer is rolled off, but the tweeter isn't. So the walls of the room will reflect this uneven response back to the listening position, along with the direct possibly relatively flat frequency response. Because of the abruptness of this change, it colors the overall sound and draws attention. And in this rather typical case, it does it in an area of frequency where the ear is most sensitive...
Another issue is where both drivers are putting out the same signal at approximately the same time, at the crossover frequency, but are electronically shifted from each other in time, due to phase shift of the reactive components in the crossover network. This causes the maximum amplitude propagation angle to not be straight forward over this frequency range. If the drivers are mounted one above the other, this maximum amplitude "beam" will shift up and down on the Y axis as a sinewave is swept through this frequency range, which will cause the room reflections to vary. So the width in frequency of the "crossover region" is an issue. Once a driver has been attenuated by more than 6dB, it's effects are pretty minimal.
By using a 4 pole active crossover network ahead of the poweramps, you reduce this damaged region down to about a half octave (generally). With a one pole passive crossover that's been carefully calibrated, you'll have about a 2 octave "crossover region" that is likely to be less than ideal. 2 octaves right in the frequencies that the ear is most sensitive to, is sort of a big deal IMO. If the 1 or 2 pole passive crossover is not carefully calibrated, you could have 3-4 octaves of this comb filter/ moving beam situation happening. When doing a passive crossover, I consider it wise and important to run each driver separately (with the crossover circuit in place), while looking at its output with a calibrated mic and a pink noise generator source. Make sure each driver is rolling off at the frequency it is supposed to roll off at. Otherwise you could have a big overlap and not necessarily realize it. More overlap means more comb filter effects and beam movement over frequency.
There are so many other issues as well. I'm just trying to make you aware of some of the main ones here. I've been researching and building speaker systems since 1967, and I'm still learning new things about speaker system design. It eventually involves room acoustics and a study of the ear-brain mechanism. How a speaker interacts with the room acoustics is one of the most important things to know about.