If you’re not making posts that the community members want to see, then they are justified in downvoting you so that they do their part to influence the culture of the community.

The point is non-community members downvoting everything. When there are five people active in a community, five non-communists downvoting everything can seriously hurt the community’s reach even when people are browsing their subs with a scaled sort.