5opn0o30@lemmy.worldtoPrivacy@lemmy.ml•How to block AI Crawler Bots using robots.txt file
215·
4 months agoWow. A lot of cynicism here. The AI bots are (currently) honoring robots.txt so this is an easy way to say go away. Honeypot urls can be a second line of defense as well as blocking published IP ranges. They’re no different than other bots that have existed for years.
Not on AWS anymore, I though.