blog: Don Marti


the Linux device driver hacker's guide to giant Internet monopoly dominance

23 August 2019

I recently found out that Linux Journal, where I was editor-in-chief for a while, is shutting down. This might seem natural, because considering all the places you find Linux—from the largest cloud services and supercomputers to the cheapest smartphones—it makes about as much sense to have a Linux Journal as it would to have a magazine called Air Breathing Aficionado. For what it's worth, MSDN Magazine is shutting down too. Are operating systems just boring commodities now?

Yes, but that's not all.

In 1994, Randolph Bentson wrote a Linux device driver for a Cyclades serial board. This piece was before my time as editor. This probably didn't seem like a big deal at the time, but it was a key event in how the Free Software scene grew into the open source software business, and then the Internet business, as we know them.

Device drivers used to be proprietary software that came on a floppy or CD-ROM with the board. What Cyclades came up with was a good early example of what Joel Spolsky later called commoditizing the complement. Eliminate many costs of keeping the software proprietary, push the maintenance programming into the open, and lower the total cost of ownership of the device. Most of the major hardware companies ended up making the same decision. Today, the operating system is an inexpensive commodity, and most hardware firms have dedicated kernel teams to keep the free part of the software/hardware combination working. This keeps the whole product (Linux plus Linux-supporting hardware) attractively priced.

The Linux business is built on ruthless commodification, and would not be a business without it. Tim O'Reilly pointed out that it's a good example of Clayton Christensen's "Law of conservation of attractive profits." Red Hat was the survivor of a crowded Linux distribution market largely because of its committment to work upstream and offload as much code review, testing, maintenance programming, and version control as possible. The OS market is a tiny fraction of what we thought it would be based on the way things were headed in 1998. Yes, individual Linux developers are well-paid, but the OS business? The ubiquitous OS license price, on both client and server, is $0.

An even bigger commodification shift came later, when server hardware became a commodity too, driving all the attractive profits to the service business. Big Internet companies as we know them grew out of the Linux scene, where the mandate to commoditize everything that you don't directly make money from is so obvious that people rarely even discuss or acknowledge it.

Commoditizing the operating system was only the first few levels of the game. Google, for example, beat level one of the game by installing racks of janky-looking Linux boxes instead of proper servers with licensed Solaris or whatever on them, then beat level two by commoditizing the mobile OS. But it's not just Google, and it's not just OS and hardware. Today's tech business is not so much about technology any more. It's more about applying the principle of commodification, learned from Linux, to everybody else's work, whether by investing in building network effects to take over services formerly run by local businesses or enforing arbitrary rules on video creators while aggressively using recommendation algorithms to drive users to "fresher" content.

The commodification play in web publishing is to control the data on who is looking at something, in order to drive the profit out of where they look at it. This doesn't necessarily work so well, but whether or not tracking-based ads work better isn't the point. They only have to work well enough to drive the web content business into the commodity category with the cover bands from Amazon Prime Music. The only real opposition comes from publishers and privacy developers. Privacy developers don't want users followed from one site to another, and publishers don't want their audience's eyeballs sold somewhere else.

The optimistic view is that better privacy in the browser will help us beat commodification. If everything works out just right, privacy in the browser means that nobody can get trustworthy data about ad impressions on random sites, which means no more infinite online ad inventory, which means that advertisers have to board the flight to quality to sites known to be trusted by their users. Then increased market power for those publishers means more expensive advertising, which means more signaling power for brands. Signaling power, if used right, builds brand equity, which means brands can spend more on ads, so they increase signal by contending to support obviously expensive content. This effect is responsible for the kind of ad-supported media that's worth real money offline, so let's make it work for the web too.

But what about the low bid problem and the crappy ad problem?

Advertisers bid less for ad impressions without tracking data when impressions with tracking data are available. According to one Google study, Based on an analysis of a randomly selected fraction of traffic on each of the 500 largest Google Ad Manager publishers globally over the last three months, we evaluated how the presence of a cookie affected programmatic revenue. Traffic for which there was no cookie present yielded an average of 52 percent less revenue for the publisher than traffic for which there was a cookie present. Lower revenue for traffic without a cookie was consistent for publishers across verticals—and was especially notable for publishers in the news vertical. For the news publishers in the studied group, traffic for which there was no cookie present yielded an average of 62 percent less revenue than traffic for which there was a cookie present.

The paper Consumer Privacy Choice in Online Advertising: Who Opts Out and at What Cost to Industry? by Johnson, Shriver, and Du finds 52% less revenue for impressions to users who deliberately opt out using an industry site.

The crappy ad problem is related. If the ad network doesn't know that you're an affluent car shopper, you're not going to get the professionally shot photo of a BMW on a scenic road. Instead, you're going to get ads for FREE* LIVER FUNGUS CURE!!!1! (just pay shipping, order auto-renews weekly).

These two effects are visible when impressions with and without cookie data, reaching a similar pool of people, are available in the same market. It would, however, be unrealistic to extrapolate from this to get to the conclusion that the result of protecting a large fraction of an audience will be that ad budgets intended to reach those people will go down by 52%. Jonathan Mayer and Arvind Narayanan point out that after GDPR, The New York Times cut off ad exchanges in Europe, and kept growing ad revenue, and that another study showed only a 4% revenue boost from behavioral tracking. (Measuring normal RTB bids against artificially cookie-blocked RTB bids seems like it would not detect a flight to quality.)

As privacy protection gets better, sites will have options to fix the low bid problem without commoditizing the content site by leaking user data. Context-based ad placement technology is still catching up to user-tracking-based technology.

We now return to the Internet Optimism already in progress

Nobody wants to be stuck being the commodity, and with decent privacy in the browser, the content site doesn't have to.

That's the basis for cooperation between privacy-protecting browsers and sites trusted by their readers. Former Mozilla COO Denelle Dixon writes, on the Digital Content Next site, In short, behavioral targeting will become more difficult, but publishers should be able to recoup a larger portion of the value overall in the online advertising ecosystem. This means the long-term revenue impact will be on those third-parties in the advertising ecosystem that are extracting value from publishers, rather than bringing value to those publishers.

I have been talking and writing about the alignment of interests between privacy developers (who don't want their users' activity from one site following them to another site) and publishers (who don't want to leak their audience data) for quite a while. But privacy and publisher market power are two parallel causes, not one big movement. The commoditizers have a lot of skill and time to put into splitting the alliance that puts publishers on the same side as privacy developers. Can the Internet ad duopoly do something to satisfy privacy demands from users and regulators without ceding market power to trustworthy sites? Two proposals.

  • Fraud Resistant, Privacy Preserving Reporting Using Blind Signatures is a system to prove that a likely human visited a site, without giving the advertiser enough information to show that a certain user visited a certain site.

  • Federated Learning of Cohorts(FLoC): We plan to explore ways in which a browser can group together people with similar browsing habits, so that ad tech companies can observe the habits of large groups instead of the activity of individuals. Ad targeting could then be partly based on what group the person falls into.

If they can't track users individually, they'll still try to figure out a way to get high-value ad impressions from known human eyeballs at random sites, and commodify publishers that way. But that depends on getting the privacy developers to decide to be fine with this kind of scheme.

The good news is that privacy developers tend to be generally sympathetic to the publishers, because positive externalities and stuff. But we're still facing the risk of privacy-acceptable but anti-publisher user data handling schemes. Privacy developers need help to keep any new privacy technology aligned with the interests of whatever publishers their users choose to trust. That means we have to commit to more ongoing coordinated open source development, with publishers who want to stay out of the precariat using and testing the same code as browser and tool developers who want to keep their users safe.

The other good news, now that I think about it, is that they're now paying the privacy/publisher alliance the ultimate compliment, by trying to split it. We're on to something here.

Continued: Open source businesses, meet the real world