The MailMerge function relies upon OLE.
Here is an IBM DeveloperWorks Blog on how the OLE "Excel.Application" component integration works in Notes.
http://www.ibm.com/developerworks/lotus/library/notes-ole/index.html
Effectively, Notes LotusScript, similar to Macros written in VB or VBA, can access properties and call functions on these OLE applications which are spawned in a child process. Notes keeps a pointer to the application in a LotusScript 'Variant' variable, and the process should be quit once it is no longer needed.
APIs for the OLE functions are quite readily available, and easily translated for LotusScript from VB.