minus-squareProntomomo@lemmy.worldcaketoSelfhosted@lemmy.world•Cloudflare blocking AI crawlerslinkfedilinkEnglisharrow-up5·3 days agoI don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling linkfedilink
I don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling