TXT

Google says it will support a shared opt-out mechanism through robots.txt that lets websites block their content from being used for AI training. The proposal would extend the long-standing web crawler standard, letting site owners set one rule to cover AI training use rather than managing separate per-company settings. The announcement is drawing attention from publishers, developers and AI firms weighing how content licensing and crawl controls should work in practice.

The tech magazine iX reports on a proposed update to the robots.txt standard that would let website operators block AI training crawlers with a single entry, instead of listing each bot separately. The idea has sparked discussion among developers and privacy advocates, who see it as a simpler way for publishers to defend their content against unwanted use by AI companies.

Google and Bing have announced they will honour a shared robots.txt signal that lets publishers opt out of having their content used to train AI models. The move comes as a study reported AI Overviews driving a click-through rate of just 0.9%, and as Bing tested ads within AI answers. Publishers concerned about losing traffic to AI-generated results are watching closely.