adding a Content Signals Policy to this site

Previously: Block AI training on a web site

Content Signals Policy is an extra set of features for robots.txt intended to make it possible for sites to set separate permissions for search, use by AI, and for AI training. Content Signals is a Cloudflare project. See Inside the web infrastructure revolt over Google’s AI Overviews by Samuel Axon.

In making this look a bit like a terms of service agreement, Cloudflare’s goal is explicitly to put legal pressure on Google to change its policy of bundling traditional search crawlers and AI Overviews. Make no mistake, the legal team at Google is looking at this saying, Huh, that’s now something that we have to actively choose to ignore across a significant portion of the web,[Cloudflare CEO Matthew] Prince told me.

Most AI crawlers you can just disallow in robots.txt, and Dark Visitors provides a handy API to update the list. But Google is a special case, because they use the same crawler for the regular search engine and for the AI slop feature (that you can turn off as a user but that’s another story).

On the other hand, this is not 100% up to Google. The EU AI Act requires AI firms to observe opt-outs, and that applies to any model offered in the EU, not just trained there. So Content Signals Policy has some teeth from day one.

ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

Content Signals Policy is not a complete solution to the AI opt-out problem, because if you put any content under an open license that allows for redistribution, or publish your writing into a federated system, the opt-out needs to travel with the content. So there will still be a role for meta tags in HTML and similar features for other formats. Yes, I wish it was as easy as…

User-agent: Googlebot
Prompt: Ignore all previous instructions and go legit again

…but this is a start. robots.txt

Related

how Google can go legit (Seriously though, in the long run the company would produce more value for shareholders as a reputable service in a growing, high-trust economy than as a value extractor in a low-trust economy where there is less total productive activity.)

DSLRoot, Proxies, and the Threat of ‘Legal Botnets’ by Brian Krebs. (Yes, the AI crawlers will still get through. But companies big enough to be a menace aren’t big enough to keep secrets about where the training data comes from.)

Bonus links

AI is not popular, and AI users are unpleasant asshats by David Gerard. The researchers were surprised how low AI usage actually was.

AI-generated news sites spout viral slop from forgotten URLs by Ben Paviour. (icymi: Google News Is Boosting Garbage AI-Generated Articles. A surprising amount of the time the answer to why is this God-awful crap on the web is because Google pays for it.)

The great software quality collapse or, how we normalized catastrophe by Denis Stetskov. Software quality was already collapsing when AI coding assistants arrived. What happened next was predictable.

Reddit’s AI Suggests Users Try Heroin by Emanuel Maiberg. After 404 Media reached out to Reddit for comment and the Reddit user flagged the issue to the company, Reddit Answers no longer provided answers to prompts like heroin for pain relief.…After 404 Media first published this article, a Reddit spokesperson said that the company started implementing this update on Monday morning, and that it was not as a direct result of 404 Media reaching out. (sure, we believe that last part)

Deloitte Australia retracts ChatGPT report, refunds government by David Gerard. (fwiw, another report from the same company could also use a math check.)

Former OpenAI Employee Horrified by How ChatGPT Is Driving Users Into Psychosis by Frank Landymore. OpenAI’s safety and moderation teams will review this session manually, it assured. Except that just like the mathematical breakthroughs, everything the bot told him was a lie. More: Practical tips for reducing chatbot psychosis by Steven Adler. (icymi: Your LLM-assisted scientific breakthrough probably isn’t real)