For a comparison of the different types of REST services in the extlib, see this document:
http://www.openntf.org/Projects/pmt.nsf/36B7CD129ED7357A86257AC6005523E7/%24file/Extension%20Library%20REST%20Services.pdf
The above document compares the following technologies:
Some of the details in sections 4 & 5 might be a little out of date, but I'm posting it now because I don't think custom services are documented anywhere else. If you have comments on the documentation please let us know.
Also, you can find Domino mail service documentation here:
http://www.openntf.org/Projects/pmt.nsf/E1B347B6A993144186257AC6005577F2/%24file/Domino%20Mail%20Service.pdf
This is the same file included in extlib releases since March 2012.