Recesses for flush mounting speaker drivers are best made by a router.
There are 'palm routers' for sale on ebay and Amazon at a low price, about £35, and router jigs to fit are available for about £15, so I had a go.

I did find two problems which were fixed as below.
First, the vertical movement is not positively fixed but only held by friction. It is necessary to hold the device by the grip (low down on the body) to avoid pushing the cutter too deep. Even so, and with the clamp tightened hard, I still found (in my testing) that the depth could wander. This was fixed as below.
This shows a short length of dowel resting on a screw head at the right and engaging with the edge of the slide, fixed in place with insulating tape. I used one each side. Worked a treat, no movement of the slider.
Next problem was that the jig did not go small enough for the 176mm diameter of the bass speaker unit. See below.
See the black slider in the track. At its left hand end I made a 3mm hole which you can see; it means the left hand clamp screw cannot be fitted but I had no problem. This hole allowed the 176mm diameter recess to be cut.
For the tweeter, much smaller diameter:
In the above photo, at the bottom right corner of the base plate you will see another 3mm hole I made. I measured the position knowing I would get close but not right. No problem; as you see I fitted only two of the plate fixing screws which allowed the plate position to be adjusted slightly. It was then necessary to drill through this hole and through the plastic above it to allow the peg to be inserted from above. It all worked.
OK, not elegant engineering by any means, but the results were excellent and the price was reasonable.
I hope you find this useful.
There are 'palm routers' for sale on ebay and Amazon at a low price, about £35, and router jigs to fit are available for about £15, so I had a go.

I did find two problems which were fixed as below.
First, the vertical movement is not positively fixed but only held by friction. It is necessary to hold the device by the grip (low down on the body) to avoid pushing the cutter too deep. Even so, and with the clamp tightened hard, I still found (in my testing) that the depth could wander. This was fixed as below.
This shows a short length of dowel resting on a screw head at the right and engaging with the edge of the slide, fixed in place with insulating tape. I used one each side. Worked a treat, no movement of the slider.
Next problem was that the jig did not go small enough for the 176mm diameter of the bass speaker unit. See below.
See the black slider in the track. At its left hand end I made a 3mm hole which you can see; it means the left hand clamp screw cannot be fitted but I had no problem. This hole allowed the 176mm diameter recess to be cut.
For the tweeter, much smaller diameter:
In the above photo, at the bottom right corner of the base plate you will see another 3mm hole I made. I measured the position knowing I would get close but not right. No problem; as you see I fitted only two of the plate fixing screws which allowed the plate position to be adjusted slightly. It was then necessary to drill through this hole and through the plastic above it to allow the peg to be inserted from above. It all worked.
OK, not elegant engineering by any means, but the results were excellent and the price was reasonable.
I hope you find this useful.
Those devices look almost designed to put you in A&E - two handle proper routers are designed that way for good reason, both hands are kept safe and when it catches you have a chance of keeping control of it.
It is small, light and easy to control. In my testing and the final work I never had any such concern, but you are right that care is needed with machinery.
Routers typically run 10 times faster than a drill, so 100 times the stored mechanical energy, meaning they take a long time to spin down and thus are completely unforgiving of any carelessness or undue haste, and have way more kick-back than you might assume from using other power tools (they can turn pieces of wood into high speed projectiles too, creating hazards for those in the vicinity).
Don't be deceived by the ease of control, if it kicks back it'll be faster than your reaction time and could simply jump out of your hand (especially if it is light weight) - two handed power grip is essential for taming beasts like this - adopt strong stable posture and pre-tense your arms when using - assume it will kick-back and be ready.
And always physically unplug from the wall before changing bits - too many people have had serious injury by ignoring that rule.
Don't be deceived by the ease of control, if it kicks back it'll be faster than your reaction time and could simply jump out of your hand (especially if it is light weight) - two handed power grip is essential for taming beasts like this - adopt strong stable posture and pre-tense your arms when using - assume it will kick-back and be ready.
And always physically unplug from the wall before changing bits - too many people have had serious injury by ignoring that rule.
Because I'm in the Ryobi camp (buy one item, charger and batteries and you're pulled into the vortex) I purchased the Ryobi palm router to supplement my router collection. The Ryobi, like most palms, is tippy. The depth adjustment isn't overly smooth, but once set and locked, it doesn't move. Because Ridgid and Ryobi are sister companies I purchased the acrylic baseplate, made for the Ridgid palm, from Lee Valley, and it fit. Having two handles on the new baseplate made using the Ryobi palm a joy. I don't use it often, but it makes it much safer and more precise.