Austria moved in with Germany during the Anschluss thing as well, I guess it depends on the level of freedom the other country has? Colonies are often left to manage themselves for the most part, with the exception of taxes and just generally doing what the sovereign nation tells them to do (I think?). Distance might play a part as well, I guess.

I kind of like the idea of the Japanese Empire all staying together, mainly because Hong Kong would have made it his duty to make Japan's life a living hell, but I guess it depends.