Did Vote Leave’s overspending cause their victory?

This article was originally written for The Spectator. I will reproduce it here, with some annotations, further discussion and personal reflection.

An Oxford professor’s claim to the High Court that it was “very likely” that overspending by Vote Leave swung the referendum for Brexit took off like wildfire yesterday. Professor Philip Howard’s analysis made the front page of yesterday’s Independent under the headline: Illegal Facebook spending ‘won 2016 vote for Leave’.

Do the numbers behind the headline add up?

Prof Howard, director of the Oxford Internet Institute, calculated that:
• Around 80 million Facebook users saw the Vote Leave campaign ads on social media during the period of excess spending;
• 10 per cent of users clicked through;
• 10 per cent of those users switched their vote as a result, giving over 800,000 switched voters.

This argument falls at each hurdle.

An obvious point is that 80m Facebook users exceeds the entire UK population. This error appears to have been caused by Prof Howard adding together daily users reached by Vote Leave. The posts are likely to have been seen by the same people across different days, meaning the total reach would be much smaller than 80m.

The numbers also wrongly conflate Vote Leave’s boosted paid reach with people organically sharing its messages on Facebook. Organic sharing costs Vote Leave nothing, meaning it could not have been a part of their overspend.

It is also worth asking another question: how many voters can Facebook reach? Recent research by the British Election Study team suggested that 55 per cent of Brits are on Facebook; Twitter is substantially less popular, with only 19 per cent of Brits using the site. If you assume that 55 per cent of the 46m people registered to vote in Britain were on Facebook at the time of the referendum, an audience of 25.6m electors could have been exposed to the adverts.

The second claim, that 10 per cent of users clicked through on the advert, also seems dubious. An estimate for the UK in 2016 put the figure of those clicking on digital ads at 0.5 per cent. In the United States, that figure is estimated at an average of 0.9 per cent. This is based on impressions — rather than users.

There is some debate about the direct success of display advertising which, at the very least, calls into question the figures behind Howard’s claim. Typically, display advertising is used to build presence and awareness, supporting other campaigns.

Thirdly, suggesting that 10 per cent of people who clicked through on these ads changed their vote from Remain to Leave is a large stretch. This study of targeting and political adverts suggests the effects of political advertising are ‘small’ and ‘often conditional’.

By contrast, causing 800,000 voters to switch from Remain to Leave in ten days would be fantastically effective. Influencing 3.1 per cent of electors means outperforming canvassing or phone calls for getting out the vote. For a digital comparison, an intrusive social media banner encouraging people to vote in the 2012 US election was estimated to increase turnout by only 0.24 points.

Prof Howard’s 2005 book, New Media Campaigns and the Managed Citizen, argues we should apply a different calculation to that submitted to the High Court.

His book says to apply a one per cent click-through rate, where 10 per cent “believe” what they read; and of that, 10 per cent act. This ‘belief’ stage appears to have been omitted in the High Court submission’s final calculation.

Using these rates, this calculation turns 25.6m people into 2,560 changed votes — hardly enough to have swung the referendum for Leave, given that their margin of victory was well over a million votes.

If we share a belief in accuracy, this erroneous claim should have limited reach.

Upon Reflection

I note I dislike the portmanteau ‘Brexit’. Words are a limited resource in these articles, and such language was deemed necessary.

I equivocated — unnecessarily — when I wrote “the posts are likely to have been seen the same people”. Since the maximal UK audience (of all regular Facebook users, not just registered voters on Facebook) is about 33m, people must have seen these posts on more than one day.

“Organic sharing costs nothing” is true, but I should have expanded on network effects: people seeing boosted posts, then sharing, which then spreads. Prof Howard assumes Vote Leave’s entire reach on Facebook was due to advertising, which is not “conservative”.

Another point raised to me was that I did not sufficiently elaborate on attribution modelling. This was, in part, thinking about the audience. The effects of display and social media advertising are not simply judged by how many people click through. This is vaguely alluded to when I wrote about the “direct effect” of display ads and its “supporting” role.

People can see an online ad, consider the product (or idea), then make a purchase later (or undertake some other action). There was an analysis of Yahoo! advertising which suggests a 5% increase in sales due to digital advertising, with 78% of that increase coming from people who never clicked on the ad itself. There was a user-based click-through rate too: of 7.2%, over the weeks of the campaign.

That analysis also raises the issue of experimental design when analysing advertising efficacy. Looking at cross-sectional estimators and differences-in-differences demonstrates a large negative bias — three times larger than the estimated experimental effect.

How you weigh impressions is important for analysing digital advertising campaigns. Attribution is also critical for multi-channel campaigns.

There was some consternation over my description of the claimed effect as “fantastically effective”. However, the article does not say — and was never intended to say — Vote Leave’s entire digital campaign could not have had such an effect. I focused on the claim that Facebook ads brought solely through Vote Leave’s overspending would be that effective. Otherwise, the counterfactual becomes Vote Leave not running a digital campaign at all.

Perhaps, I chose the wrong adjective as emphasis: I meant “very”. I also considered undertaking some British understatement with “rather”. In his High Court submission, the counterfactual proposed by Prof Howard was: Vote Leave did not advertise on Facebook for ten days. If Facebook advertising alone over the course of those ten days induced a shift equivalent to 3.1% of all electors on Facebook — not some targeted subgroup, but every registered voter on the platform— then that would be very effective.

As stated in the article, a comparison could be made against the effectiveness of Get Out the Vote campaigns on voter turnout:

We should think about multi-channel campaigning. People see and hear campaign messages on television, in newspapers, on radio, from canvassers, on leaflets, on general websites, on social media, and from friends and family.

Even in wholly digital journeys, attribution is a difficult exercise: how people vote is not simply a function of how many ads from each campaign they see. What effect should we reasonably ascribe to some sponsored Facebook posts and display ads, versus watching campaign broadcasts, receiving leaflets and getting a visit from canvassers?

Another point I raised in the original version — which got cut during sub-editing — was the relative stability of public opinion. In published opinion polls, the average Remain share (after excluding undecided people) never diverged much from 50% in the final four weeks.

This is from Prof Curtice’s presentation to the NCRM meeting, which I attended. (Source: NCRM)

YouGov’s data from their internet panel suggested the peak in Leave support occurred on 13th June — ten days before the referendum.

A daily estimation of public opinion of the EU referendum, adjusted for house effects, suggested Leave were ahead throughout the entire formal campaign.

The house adjustments mean the final result comes in at the correct level. (Source: LSE)

Political campaigning can be effective — but there is more than one campaign, and contact may not directly persuade. The claim (which I sometimes heard in response to my posts on Twitter) there was a large and demonstrable shift towards Leave in the final days is not observed in polling data: the net movement was towards Remain. This should induce caution when surveying for plausible campaign effects.

The effect size I was citing referred to the 2012 US election, whilst a similar Facebook campaign was run in 2010. Erroneously, I wrote “the 2010 election”, and corrected the article later.

One broad concern I have after writing the article is that I did not adequately convey my knowledge. I volunteer as a Statistical Ambassador for the Royal Statistical Society, but I work as a digital insight analyst. In the past, I have analysed digital marketing campaigns, and compared attribution models. (My current role is focused on improving digital journeys, rather than marketing analysis.) Having someone on Twitter attempt to explain to me how Facebook ads work was simultaneously funny and noxious.

The final sentence of the article was the most important to me. It explains why I wrote the article. I hope I get the opportunity again, and I have considered the responses I received.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.