I built an A-40 and use it every day on speakers of my own and I'm happy with it.
the point is:
1) Do you need 40W per channel on headphones ???
2) Don't forget you will get the noise associated with the PSU requied to feed a 40W class-A amp. In my sense, maybe too much noise to be a good thing.
Why don't try the nice SDS Labs Headphone amp. I drive a pair of 32 Ohms Grados with it. Sounds very good and dead quiet at idle...
I do want to use it for normal speaker (dynaudio diy) but I also want a headphone connection just to use when my girlfriend is watching telly or something
the A-40 is a power amp and you barely need 1X or 2X line level (ie 2-4V RMS). A well buffered opamp based design should cut mustard like a purpose built mustard cutter for you. If I was you, I may be looking elsewhere that modifing the A-40, at least for the purpose.
Resistors in series with the speaker outputs is all you need. Value depends on your headphone impedance. Try 100 ohms to start if you have low-impedance (32 ohm) cans.
Using a dedicated headphone amplifier may get you better sound, but it is not necessary.
paulb answer seems to be something really acceptable but I wonder if you really get rid of the noise in this case.
For the SDS Labs little amp, I have had the print made for me for this project. Just check if the pcb transformer holes are at the right distance from each other. I remember I've had some problems with that... IRF610 and 9610 could be a better pick from their lower gate capacitance but IRF620 and 9620 are making great music in mine.