hylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 2 days agoBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersjlai.luimagemessage-square77fedilinkarrow-up1568arrow-down15file-text
arrow-up1563arrow-down1imageBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersjlai.luhylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 2 days agomessage-square77fedilinkfile-text
minus-squareAHemlocksLie@lemmy.ziplinkfedilinkEnglisharrow-up16arrow-down1·1 day agoPretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?
minus-squaretomjuggler@lemmy.worldlinkfedilinkEnglisharrow-up4·7 hours agoYes, CloudFlare blocks agents completely if they ignore it’s restrictions. The key is scale - CloudFlare has a birds eye view of traffic patterns across millions of sites and can do statistical analysis to determine who is a bot. I hate the necessity but it works
minus-squareSv443@sh.itjust.workslinkfedilinkEnglisharrow-up9arrow-down1·1 day agoLike a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys
minus-squareFreedomAdvocate@lemmy.net.aulinkfedilinkEnglisharrow-up1·7 hours agoCloudflare definitely can and does stop the vast majority of actual professional bad guys.
Pretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?
Yes, CloudFlare blocks agents completely if they ignore it’s restrictions. The key is scale - CloudFlare has a birds eye view of traffic patterns across millions of sites and can do statistical analysis to determine who is a bot.
I hate the necessity but it works
Like a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys
Cloudflare definitely can and does stop the vast majority of actual professional bad guys.