Differences between single-section and two-section filters
 

The cost of a capacitor is not that expensive but why there are single section filter (LC) and two section filter (CLC) designs? What are their individual advantages?

If you are talking about an EMI filter a 2 section LCLC filter may be smaller than a 1 section filter, because the roll off (attenuation) vs. frequency is higher.

I am referring to the choke being used in power supply. Some circuit connects the choke to the rectifier without a cap and some has a cap. There is a voltage output difference but other than this, is there any particular reason to use each of them?

Forward converters as well as half/full bridge and push pull must not have a capacitor on the rectifier side of the inductor as the inductor is part of the voltage regulation circuit.
Flyback converters, however, must have a capacitor on the rectifier side. The LC filter afterwards is used to reduce noise. In an application that does not mind much noise (such as charging batteries or running motors), the extra filter can be omitted.