Uh, over one year later.
But I think I found it:
In the SSJS script library xpTagCloudServer you should change the line 24:
minEntryCount: null, // if null, tune the number based on the # of docs in the view
Actually I changed it to "minEntryCount: 1", because a category without any entry isn't interesting.
There could be some caching which would prevent you from seeing the effect immediately.