1- getComponent('dialog2') is returning null because dialog2 is not a component inside dialog1. And this is not handled by the getComponent() method. The workaround is to use a Java method from FacesUtil instead
2- There is a race condition between the close dialog request and the open dialog2 one. I don't understand all the details, but when you open the dialog2 later then it works. The workaround is, for example, to do it after 500ms.
Bellow is a modified function that makes it works. Note that it requires 853 as the postScript method ia a new addition to this release.
That said, chaining the dialogs like this is generally considered as a bad user experience.
function closeDialog1AndShowDialog2() {
var dialog1 = getComponent("dialog1");
dialog1.hide("dialogPanel");
var dialog2 = com.ibm.xsp.util.FacesUtil.getComponentFor(view, "dialog2")
view.postScript("setTimeout(function() {XSP.openDialog('"+dialog2.getClientId(facesContext)+"')},500);");
dialog2.show();
}