24 Jun 2024 Search

Quick test of the perplexity.ai thing. From here it looks like they ignored this site’s 3 layers of AI protection:

  • robots.txt
  • X-Robots-Tag HTTP header
  • noai meta tag

and that they’re crawling from three different IP addresses with three different user agent strings, which look like normal browsers.

The test I did: I have some pages on this site that are not linked to from anywhere, so tested to see if this company would crawl one. (Mostly just random stuff that I have shared with friends but not blogged.)

My perplexity.ai query:

Can you explain how to play the board game [game name] which is covered by the ruleset at [URL]?

and that URL (which gets a couple of hits on game nights, that’s it) was immediately visited by not one, not two, but three different IP addresses.

44.221.181.252 “Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36”

45.56.133.241 “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36”

134.73.83.233 “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36”

Why three? The first reason I can think of is to try to evade a block, but really? Any better explanations?