Block AI training on a web site

robots.txt

Known Agents is a good source of an up-to-date set of lines to add to your robots.txt file.

This site uses the API to catch up on the latest. So if I fall behind on reading the technology news, the Makefile has me covered.

# update AI crawlers blocking list
# FIXME: use new domain
tmp/robots.txt :
        curl -X POST "https://api.darkvisitors.com/robots-txts" \
                -H "Authorization: Bearer $(shell pass darkvisitors-token)" \
                -H "Content-Type: application/json" \
                -d '{"agent_types": ["AI Data Scraper", "AI Assistant", "Undocumented AI Agent", "AI Search Crawler"], "disallow": "/"}' \
        > $@

# The real robots.txt is built from the local lines
# in the conf directory, with the fetched lines added
public/robots.txt : conf/robots.txt tmp/robots.txt
        cat conf/robots.txt tmp/robots.txt > $@

One of my cleanup scripts gets rid of the tmp/robots.txt if it gets stale, and I use Pass to store the token.

robots.txt continued: Content Signals Policy

This is for crawlers that have multiple purposes and, for whatever reason, you don’t want to block entirely. Content Signals Policy is a new standard that we don’t know whether or not Google will follow. Update the robots.txt section for those crawlers to include a Content-Signal line, like this:

User-agent: Googlebot
Content-Signal: search=yes, ai-input=no, ai-train=no

There is also a block of verbiage to add to the top of the robots.txt file, available from the Content Signals site. More at adding a Content Signals Policy to this site. I don’t have any pages disallowed for Googlebot entirely, because I am a big Internet optimist. Cloudflare CEO Matthew Prince said, Inside of Google, there is a fight where there are people who are saying we should change how we’re doing this, and I agree that Google can still go legit again someday.

X-Robots-Tag HTTP header

DeviantArt covers how to set the X-Robots-Tag header (which also has other uses to help control how search engines crawl your site) to express an opt-out.

On Apache httpd (I know, I’m old school) it’s something like this:

Header Set X-Robots-Tag "noai"

You can check it under “network” in browser developer tools. It should show up in response headers.

noai meta tag

Raptive Support covers the noai meta tag. Pretty easy, just put this in the HTML head with any other meta and link elements.

<meta name="robots" content="noai, noimageai">

That support FAQ includes a good point that applies to all of these—the opt out is stronger if it’s backed up with the site Terms of Service or User Agreement. Big companies have invested hella lawyer hours in making these things more enforceable, and if they wanted to override ToS they would be acting against their other interests in keeping their sites in company town mode.

new: privacy opt out for servers

This is the first site to include the new SPC meta tag and X-Robots-Tag header for a privacy opt-out that works like Global Privacy Control but for servers. Basically you have legally enforceable rights in your personal information, blogs have personal information, but regular GPC only works from your browser (client) to company on the server. This goes the other way, and sends a legally enforceable* *yes, I know, this has not yet been tested in court, but give it a minute, we’re just getting started here privacy signal from a personal blog on the server to an AI scraper on the client side.

So the new header on here is

X-Robots-Tag: noai, noimageai, SPC

So we’re up to five, somebody send me number six?

Related

Block GenAI Bots with Eleventy – Pixievolt No. 1 Blog covers how to set up the robots.txt updating with the Eleventy static site generator.

Google Chrome ad features checklist covers the client side of this— how to protect your personal info, and other people’s, from being fed to AI (among other abuses)

remove AI from Google Search on Firefox: how to remove the “AI”-generated material from Google search results

Block the Bots that Feed “AI” Models by Scraping Your Website by Neil Clarke

How to Stop Your Data From Being Used to Train AI | WIRED covers much other software including Adobe, Slack, and others. The list below only includes companies currently with an opt-out process. For example, Microsoft’s Copilot does not offer users with personal accounts the option to have their prompts not used to improve the software.

opt out of Google Page Annotations is yet another problematic Google feature to opt out of

Bonus links

The Internet is a Series of Webs The future of the internet seems up in the air. Consumed by rotting behemoths. What we have now is failing, but it is also part of our every-day life, our politics, our society, our communities and our friendships. All of those are at risk, in part because the ways we communicate are under attack. (So if Google search ads are scammy enough to get an FBI warning, Meta is a shitshow, and Amazon is full of fake and stolen stuff, what do you do? Make a list of legit companies on your blog and hope others do the same?)

For tech CEOs, the dystopia is the point The CEOs obviously don’t much care what some flyby cultural critics think of their branding aspirations, but beyond even that, we have to bear in mind that these dystopias are actively useful to them.

Apple Removes Nonconsensual AI Nude Apps Following 404 Media Investigation (think of how bad the Internet would be without independent sites covering the big companies…then go subscribe to 404 Media.)

Amazon is filled with garbage ebooks. Here’s how they get made. The biographer in question was just one in a vast, hidden ecosystem centered on the production and distribution of very cheap, low-quality ebooks about increasingly esoteric subjects. Many of them gleefully share misinformation or repackage basic facts from WikiHow behind a title that’s been search-engine-optimized to hell and back again. Some of them even steal the names of well-established existing authors and masquerade as new releases from those writers. (I’m going to the real bookstore.)

“Pink slime” local news outlets erupt all over US as election nears Kathleen Carley, a computer science professor at Carnegie Mellon University, said her research suggests that following the 2022 midterms “a lot more money” is being poured into pink slime sites, including advertising on Meta.