16 Apr 2026 Search

First, I used to assume that if the RSS feed is valid, the links are good.

But there’s the possibility of a “works on my machine” feed. Not going to mention a specific company here, but I spotted a feed on an HTTPS site but with HTTP localhost links.

    <title>Funded Startup Blog</title>
    <link>http://localhost:5150/blog</link>
    <description>Awesome technology and business insights from our innovation journey</description>
    <item>
      <title>Our thing works with some other company's other thing</title>
      <link>http://localhost:5150/blog/please-googlebot-dig-these-keywords</link>

See the problem? The links work fine when they preview on localhost with the dev server running, but on the real Internet, not so much. And the feed is technically valid, just not really working.

So for now I am doing this.

def fix_url(source, dest):
    "For sites that leave localhost links in the production RSS feed"
    "Replace the scheme and netloc"
    sp = urllib.parse.urlsplit(source)
    dp = urllib.parse.urlsplit(dest)
    if dp.netloc.startswith("localhost:") or dp.netloc.startswith("127.0.0"):
        logging.info("fixing localhost link to %s" % dest)
        return urllib.parse.urlunsplit([sp.scheme, sp.netloc, dp.path, dp.query, dp.fragment])
    return dest

This is not going to be right in all cases (it doesn’t work if they use a different virtual host for the feed) but for what I have found so far it’s better than letting the localhost links be.

The second assumption is a little tougher to give up. Read The Internet’s Most Powerful Archiving Tool Is in Peril from Aram Zucker-Scharff.

Whatever you think about our machine learning overlord corporations, there’s no doubt that they are sucking immense value from journalists and giving almost none of it back. The only tool media companies really have in their arsenal is blocking.

I used to be able to assume that I could link to something from here, and if the original went away, I could swap in an Internet Archive link. For example, in Before surveillance capitalism and surveillance advertising, there was surveillance marketing I found what I think is the earliest use of the term—not by academics or privacy advocates, but by Mark Cameron in Marketing Magazine.

But now any halfway awake web publisher is going to block archives, Common Crawl, and other services, to try to keep their pages away from the deeply unsympathetic AI tycoons. (If you want a recipe for risotto, go to the library or find a legit human-tested recipe site.)

So that means if I link to something but I don’t have a local copy, it need to be an error. This way, if making an archive.org link doesn’t work, I can put up the local copy if it’s a government work or out of copyright for some other reason, or replace the link with a fair use excerpt.

I’m not going to automate it, since a lot of sites are going to be good enough at avoiding scrapers that they could avoid whatever I could set up. But now there’s one more task to add to the new page build process.