everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

  • SavvyWolf@pawb.social
    link
    fedilink
    English
    arrow-up
    1
    ·
    7 hours ago

    It’ll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else’s site.