It will be the case with the fediverse in general. But the point of the federation is to make it easier to switch instance, reducing the power of the owners.

An ideal solution to this would be a fully distributed system. But this has many technical challenges as it pushes the complexity on the client side. And the moderation is then also more complex. Federation aims at finding the good trade-off between giving power to a few capable people to manage the network, yet making it difficult for them to abuse the system as users can easily switch ship without losing their entire social network. This is similar to emails - changing your email address is a pain, but you don’t lose your contacts and can still talk to people with your new address.

This is a concern regardless of instance and the size of the instance. At least in my mind.

Personally I don’t have too much concern right now but if anything does come to light that changes that then I’ll just leave.