1st order of business is deciding what impedance load you want to work with and how much power you need; dynamic headphones range from 16 Ohm to 600 Ohm, sensitivity from slightly less than 90 dB/mW to over 100 dB/mW
HeadWize faqs points to all sorts of good relevant headphone information - the link to Rane under headphone power requirements gives the sensitivity of many headphones
I always have to point out the difference between LOUDNESS and HEADROOM - read
http://headwize.com/articles/hearing_art.htm twice, once for avoiding hearing damage from average LOUDNESS and again for the required dynamic peak HEADROOM of real music - that you should want to cleanly reproduce, but only for seconds per hour of listening
taking some dynamic range values from the headwize article you can see that while 10 mW average power into most headphones could lead to hearing damage with extended exposure, it wouldn't be totally unreasonable to want 100-500 mW peak power to reproduce natural music peak levels without clipping
Once you have load Z and peak pwr you can choose a bias current and current sink circuit as well as transistor sizes and if heatsinking is necessary