The agent (RWF DOC QUERYCLOSE) will save the document in notes client. The agent can usually be called from the queryclose event but it does not have to be.
To avoid the problem you have described, do not call the agent from the queryclose event, instead call it directly from the buttons, if you have a "cancel" button, do not call the agent and the document will not be saved.
Alternately, you can put logic in the formula for the queryclose event to not call the agent.
I think there is a sample app, that has an example of this.