Ethics & SecurityNovember 02, 2026
-38
🧊
Google Unifies AI Opt-Out in robots.txt
Google to honor a single robots.txt token blocking AI training from 2027.
#robots.txt#Google#IETF#AI training#crawlers

🔥 What happened
Google will support a unified policy token in robots.txt starting 2027, letting site operators centrally opt out of AI training. Instead of maintaining separate blocks for each AI vendor, one signal will cover all participating developers.
💡 Why it matters
Publishers previously had to block dozens of crawler tokens like GPTBot or ClaudeBot one by one – a patchwork. The new token, based on an IETF draft, is voluntary: only cooperative AI providers will honor it, and it can't technically enforce anything.
⚡ Our take
A win for publishers on paper, but a toothless tiger: anyone already ignoring robots.txt will ignore this too. Without enforcement, it's just a gentleman's agreement in the AI era.
The title, summary and analysis of this item were produced automatically by an AI system and have not been editorially reviewed. They may contain errors, bias or omissions — when in doubt, read the linked original source.
Deep Dives & Similar Intelligence

Google Slashes Video Generation Prices 30%
AI modelsNovember 05, 2026

Google's satellite writes poetry in orbit
ResearchNovember 01, 2026

Google Slashes Video AI Prices by 30%
AI modelsNovember 06, 2026

Google's Dreambeans Lands in India, But Will It Stick?
Tools & ProjectsNovember 03, 2026

Google Now Hacks Itself With AI
Ethics & SecurityNovember 03, 2026