This trope has been explored, sure enough.

Recently, I recall the comics DIE, although I didn’t finish it yet. And also the TV show The Magicians. In both cases one of the crew stays to be a king while everybody else goes home. And time runs at different rates on either sides. And then they meet again. Hijinks ensue.