If I close the configuration dialog via the save configuration button and try to re-open it in the same page, the dialog will not re-appear.

I have to open a new xpage instead... :-/