Drunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 25 days agoHow to combat large amounts of Ai scrapersmessage-squaremessage-square54fedilinkarrow-up1121arrow-down16file-text
arrow-up1115arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 25 days agomessage-square54fedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-squareSavvyWolf@pawb.sociallinkfedilinkEnglisharrow-up1·22 days agoIt’ll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else’s site.
It’ll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else’s site.