I use to use old forums, i don’t think the fediverse is worst then those old systems.
I think you could just ask a one time fee when registering or a monthly fee if you want to reduce moderators burnout or increase professionalization (in the best possible sense). maybe even just have the money used and publicly donated to some non profit (or stuff like funding lemmy development). maybe having a place where people know everyone donated to achieve some worthy goal will increase the trust between people.
If you can program you can probably create an instance and then a moderation bot that bans people with more then X comments or Y posts a day. maybe that would increase the average quality of content. sounds like an interesting experiment.