When, from the outside, a collection of people or an institution is doing things I strongly disagree with, it often turns out that from the inside I can see all the mechanisms and incentives that make perfectly normal people works towards bad outcomes. Take online advertising. In my last year at BuzzFeed I finally bit the bullet and started working in the part of the team that makes the money: the ad tech team.

Proposition: online publishing is generally supported by some combination of a paywall, donations, a benefactor, affiliate commerce, or ads. Financial Times is entirely behind a paywall. The New York Times has a partial paywall and ads. The Guardian gets by with some ads and some donations. The Washington Post is owned by Jeff Bezos who seems to have some interest in its continued existence and could pay for it to lose money indefinitely. Wirecutter and other review sites make a lot of money out of affiliate links to products, supplemented by ads. Finally, BuzzFeed and myriad other places make most of their revenue from advertising.

Every single one of those revenue models poses problems from a business perspective and an ethical perspective. My experience is with advertising though so I’m going to to walk you through what I learned about how the ecosystem actually seems to work. It’s rotten but it pays for the words we eat all day every day.

Let’s start with the problems an ad supported publisher faces.

No bullshit ads primer

revenue = impressions × price

impressions = pageviews × ad density

Advertising revenue is derived from the number of ad impressions a publisher gets multiplied by the amount they were able to charge advertisers per impression to put their ads there. If somebody sees your ad on your web page, that’s an impression. As I learned more from the people who sell campaigns to advertisers, the amount that advertisers care about impressions versus more sophisticated metrics shocked me. For all of our fussing about targeted advertising and getting the right ads in front of the right people, the bottom line is to make money from ads you need to get people on the site and they need to have your ads in their viewport, getting impressions.

The number of impressions you get breaks down further into how many page views you got multiplied by the amount of ads you managed to show the user in the time they were on the page, also known as the ad density.

Because one factor of impressions is page views, your revenue is directly affected by anything that curtails the flow of users to your website. A decade ago along came the big platforms in social media and search that started to cannibalise those page views. People discover your content through their news feeds and timelines and they tap on the articles. If you’re lucky they end up on your actual website and your banner ad loads up top and gets you an impression. If not, they see a (much easier to read) stripped-back version of your article hosted by the platform themselves and it’s little to no ad revenue for you. The firehose of traffic is controlled for the most part by Facebook, Twitter et al, and they can (and have) arbitrarily decided to point it elsewhere in the past.

Factor two of the ad revenue equation is CPM (cost per mille, as in thousand impressions). How much do you get to charge for each of the impressions you went out and got for your advertiser? A lot of factors go into pricing but high on the list are: the prominence of the ad and the audience quality. Prominence of the ad works across time and space: an ad that you see for longer is more valuable, and an ad that takes up more of your screen is more valuable. Publishers tell stories about “premium packages” and specially integrated ad experiences but ultimately, it’s about how much time and space your ad is going to take up.

Publishers improve the quality of their ad audience by targeting ads and making advertiser’s ads are going to the right people. Right people in this context often means the right age group and the right income bracket. Knowing those aspects of your audience and being able to produce segments that represent specific people the advertiser wants to reach, that’s the desire that drove the worst of the worst when it comes to web advertising.

Play the game

I worked in the ad tech space for one year, and I have absolutely no idea how the data privacy angle actually works. There are thousands of companies and products with impenetrable names who do something related to “audience insights” or “activation”. All of them want information about your audience. In exchange they offer you the ability to do things like target ads to racketball players in Utah over the age of forty.

Quite how those companies figure out who lives where and what they’re into isn’t clear to me, nor is whether the classifications are even accurate. What is obvious is that they rely on a collective effect. The more websites sign up and hand over user data, the more data points they have to join up from that user’s behaviour across multiple destinations on the web. Your profile of a user could start with matching their IP address with up with that of somebody who also has a plan with Verizon. Now you have a home address because Verizon has that. Your data connectivity platform is also used by Amazon, The New York Times, and CNN, so you also know what category of products they buy at what price points and what their key news topics are.

There is no mastermind behind it all. Each company plays it part and provides a service to a partner who makes up a different piece of the puzzle. The publishers share their data because they can increase their CPMs by offering better targeting to advertisers, the data platforms sign up as many sites as they can so they can offer a product with better targeting capabilities, the advertisers pay to place ads to market their stuff to an audience who might be interested and hey, even support digital media with some big ad buys.

Ironically, at the scales of traffic that a lot of publishers get, the kind of specificity I’ve described would yield an audience that’s too small to even make money from. Nevertheless, we want to be offer that ability to advertisers and put it in the slide decks we present to the Chief Marketing Officer of Coca Cola. A lesson I learned, and the reason why I doubted  some of the targeting capabilities offered by third parties, is that there need not be much of a relationship between the level of sophisticated targeting actually being done and the claims being made in marketing materials to other companies.

GDPR and other data privacy regulations were intended to solve this by forbidding publishers from sharing users’ data like this without their explicit and informed consent. From a part of the company shielded from the sausage factory, before I worked directly on ads, I welcomed these changes when they were introduced. I hoped that other jurisdictions would follow the EU and introduce similar measures, and so they did.

It’s incredible how quickly GDPR and other data privacy regulations go from looking like the solution to all our problems, to an implementation detail, another piece of third party code thrown into the slurry. Most publishers handled the informed consent problem by reaching out to another third party to integrate their consent management platform. Script goes on the page, pop-up appears, user smashes “Consent to cookies” as fast as they can as they slice through the thicket of bullshit in front of their news article, informed consent acquired, fire the data off to the third parties and render the ads.

Consent management platforms break, they can be implemented incorrectly. In one failure state, all the ads might disappear from the page. In another, a rare user who has never consented to a single crumb of third party cookies will have their data sent to all askers without further regard. It is unclear even to people adding a consent system to the publisher’s website, what precisely is allowed or is not allowed to be shared and with whom under the various configurations you can choose from the consent pop-up. No malicious intent is needed for a website to add your visit to their article from an iPhone in north London to the constellation of data points about you that some platform like LiveRamp has stored, even though you denied all the cookies offered to you.

I don’t think I’m shocking anybody describing the situation in online advertising. You would even be forgiven for thinking things will soon get better, for users’ data privacy at least. Browsers are increasingly sparring with this ecosystem, locking down third-party cookies, blocking requests to trackers and ad networks like good little bodyguards. The ecosystem responds. Reports of the death of the cookie are greatly exaggerated. Universal ID is a system that circumvents third-party cookie blocking by laundering user data through first-party cookies instead. The incentives for publishers to integrate with these trackers are the same as in the old scheme: if you have this tracker in the page, x giant advertiser will pay more per impression because they think they know who they’re getting with more accuracy.

People are doing their local best

Now you’ve had my very editorialised primer on how an ad-supported online publisher works. The thing I want to emphasise is that the ecosystem doesn’t require any malicious actors to produce terrible outcomes. It’s just about how the incentives are arranged.

Start with the user, so the books say. Users want to be informed, entertained, or even mollified by some content. They don’t think they should have to pay for that content really, there’s so much free stuff around and besides, there’s ads. The journalist wants to write about what they think is interesting or important for their audience. Maybe they’re not writing what they think will be their career-defining work right now but it’s not bad and they trust their editors’ instincts and the job market for reporters right now is such that they should be happy to not be freelancing. Besides, they work hard on their stories and the analytics tools the publisher provides tell them hundreds of thousands of people read their words every day.

The editor wants to be known for running a team that produces meaningful content that also performs. That is, they want content that gets clicks and is part of the conversation, slipping the greens (unique fact-based reporting and cultural analysis) in with dinner (posts that will appear in the search results of the day’s top ten search terms). They want to cultivate new writers and give their team the space to do great work, and they pray they’re not going to be pulled into a meeting room any time soon, given a dollar amount and a list of the names of people in the bullpen behind them.

The CEO wants to convince investors there’s growth coming, that they’ve got new ideas for revenue that haven’t already been tried to varying levels of success by every other publisher. The CTO wants to come up with ways they can increase revenue per user, reverse declining page views, and reduce the dependence the business has on platforms that could switch off the money hose tomorrow with an algorithm update. The ad sales team want to hit the excessively aggressive revenue targets the CEO set to wow the investors. They can’t do much about their audience size, that’s up to editorial and tech to figure out. So they focus on selling high CPM ads: big ads, highly targeted. The big clients won’t sign off on deals until their tracker of choice is in the page to verify they’re hitting the audiences the guys from sales are claiming.

The ad tech team want to be part of the solution for increasing those revenue numbers. They field those requests from sales for the umpteenth tracking script, another universal ID solution. They try and do as little harm as they possible, hitting the page performance as little as they can with the torrent of ad scripts. They try and push the ads out of the way of the content for the most part, put the kibosh on a new full-page popup ad format sales are asking for. When they do implement a new third party script, they try and decrypt the terrible documentation and ignore some of the more outrageous demands it makes of the page’s performance and users’ data.

It goes on. I could expand this circle to include the ad buyers, the programmatic ad exchanges, the product managers at the big platform companies. You get the picture though, each of the individual actors here are just doing their best for a hyper-local good. I’ll go out on a limb and say that nobody in this web has an actual in-depth understanding of the domains of their inter-connected counterparts. I can give the picture in overview. Wizened types who’ve been at it for a decade could go deep on how the data aggregators work and how they prop up the whole online publishing industry, but they don’t know the letter of the law coming out of Brussels and they don’t actually know why Facebook makes the changes to its news feed that it does.

I’ve only been talking about the ad supported revenue model, because that’s what I worked on at a company that has history had an anti-paywall stance. That leaves the other models, which each have their own foibles. There’s affiliate content (editorial conflict and biased coverage), the wealthy benefactor (implicit censorship based on the benefactor’s interests), donations (unreliable and insufficient revenue), and the paywall. I don’t go as far as my former employer and claim that paywalls are fundamentally anti-democratic but there is a kernel of truth in the notion that the truth is paywalled but the lies are free.

These are some of the pathologies in the corner of the media industry I worked in. None of them belong to arch-villains or greedy monsters. Most of them belong to the inescapable, choking pull of capital, but some are built piece-by-piece by kind and thoughtful people I variously call friends and colleagues.