Overreliance on polling is sinking American democracy
Democratic leaders: be bold and strategic, instead of making every decision based on data
Boosters for COVID-19 have become a high-stakes issue. This summer, Americans thought COVID would fade, and we would be looking at a more-normal fall. But then new variants arrived.
The story of boosters highlights a critical – and broken – aspect of American scientific and social decision making. It’s about how we use data. And it’s not just related to boosters, because data affects all sorts of liberal and left political decisions.
So how is data being used incorrectly? Some of the data presented to advisory panels about boosters has been flawed, because that data comes from polling. And polling has a big blind spot.
Here’s the example – it is about polling data that advisory boards used. In September, Stat said:
The poll presented Thursday to the CDC’s advisory panel revealed that a third of people who are still unvaccinated say the need for a third shot would make them less likely to agree to get any doses of Covid vaccine.
This is a fine thing to try to study. But making a decision not to vaccinate based on this poll has a critical flaw – the poll does not account for persuasion. This poll gave people some information, that a third shot might be needed. And it measured soon after that what those people thought. That is, by definition, this poll can only capture short-term effects.
But we know persuasion is not a short-term effect. Persuasion requires repetition, and depends on information received from other people in society. Based on what we have seen in other countries, it is absolutely possible that endorsing and promoting boosters would have had the opposite effect shown by the poll – it could well have made people more confident in the vaccine. Reasoning from short-term polls alone is limiting.
Persuasion is non-linear, and polling assumes linear effects
A hidden assumption of polling is that the effects it measures will sum linearly. For example, if talking about boosters decreases trust in the vaccine, then talking about boosters a lot will just further decrease trust in the vaccine. But that is not the way people work. Having a lot of people talk about boosters a lot might well increase trust in the vaccine. Polls have trouble measuring this. It’s a huge problem, because so much of left/liberal decision-making depends on polls.
This short-term nature is just a fundamental limitation in how polling works. To overcome this limitation, someone would have to follow the same people over time and give them lots of messages. That is possible, in theory, but it is usually prohibitively expensive to do this with a polling or survey design, and so it isn’t done.
This limitation in data has led to fundamental misunderstandings about what politicians should do. Polling studies in political science have generally found that persuasion is hard or impossible or even counter-productive. But the problem here arises from polling itself. Seeing the effects of persuasion, if they are long-term, would be very difficult with polling. Giving someone a small number of messages is very unlikely to change someone’s mind.
Studies that use natural experiments, not polling, show that major persuasion is possible
Other studies contradict this received Democratic wisdom that public opinion is fixed – and therefore, that we should do whatever polls say is the popular thing. Studies that use natural experiments find that public opinion can be shifted by huge amounts by media consumption – that is, by repeated messaging over long periods of time. This work started in economics, where natural experiment methods were pioneered. (Two economists received the Economics Nobel prize this year for natural experiment methods.) DellaVigna and Kaplan (2007) used the time Fox entered different counties to show that cable news has a large effect over years on public opinion. Similar approaches using channel position, often set by when Fox entered a market, and which affects how much viewers watch Fox at the margin, have repeatedly found similarly large persuasive effects of cable news (Yurkoglu and Martin, 2017.) These papers are all economics papers published in high-profile economics journals, and yet the effects of media persuasion shown in these studies, until recently, seems to have made little impact on political scientists – and on politicians.
Some good news is that a few political scientists are starting to adopt natural experiment methods. They are also finding (Warshaw et al., 2021) that persuasion is possible.
And a recent study by some political scientists (Foos and Bischoff, 2021) , using economics-style natural experiment methods, confirms that media can have huge persuasive effects.
A major takeway is that we now have is two lines of work that show that Murdoch media can create big changes in public opinion: the channel position studies and Foos and Bischoff. One line of work studies cable news, the other studies a tabloid. One uses instrumental variables, the other difference-in-differences. One studies American voters, the other British voters. And yet despite these major differences in approach, they each find very large persuasive effects of Murdoch media. Replicating a result like this with different methods gives us extra confidence in the result. Which means that despite skepticism from Democratic consultants, media matters! It’s not just that media gives people what they want to hear, it’s not just that media fills an niche and satisfies demand. No: media changes minds.
Natural experiment:
Some event in the world that allows researchers to divide individuals up into experimental and control groups without actually performing a randomized study. An example is the boycott of the Murdoch tabloid The Sun in Liverpool due to The Sun’s toxic coverage of the Hillsborough soccer disaster. This boycott meant people in the boycott area weren’t exposed to The Sun, naturally sorting people into a group exposed to The Sun (outside the area) and those that were not. No survey was needed to determine the effect of the messages the Sun conveyed.
Foos and Bischoff (2021) used this natural experiment to show that the Murdoch paper had a huge effect on Leave vs. Remain opinion. In other words, The Sun changed people’s minds. It persuaded them.
Media persuades people, but Dems can do persuasion without a Fox-for-Democrats on their side
The huge persuasive effects found in the studies above result from the influence of media, and Democrats are not likely to create instantly a version of Fox for Democrats. So some Democratic leaders might feel like they should give up on persuasion, and just keep going with their poll-tested, limited, kitchen-table messaging.
That would be the wrong conclusion. Democratic leaders should look instead to the effects of Benghazi hearings, or of “but her emails”, or Whitewater, or of critical race theory. Republicans repeated those messages, and they worked. They persuaded people.
The most clear example of Democrats’ failure to do persuasion because of overreliance on polling is Trump impeachment. In 2019, polls showed that “the idea of removing Trump from office was deeply unpopular.” From the linked article:
Consultants working on individual races recalled seeing some polls in targeted districts where voters opposed impeachment by 40-point margins. When Democrats won the House, despite a flood of anti-Pelosi ads, DCCC leaders cautioned the new majority that Trump’s alleged Russian collusion was too politically tricky to navigate in too many districts, said operatives familiar with the warning.
But this is polling data! It measures the response to a single question. It does not have the ability to measure what would have happened if Democrats had run months of hearings, used inherent contempt, and dragged Trump’s corruption into daily news cycles.
Indeed, as soon as Americans learned that Trump was corrupt due to the Ukraine shakedown, “the polling swing in favor of impeachment happened faster than even its loudest boosters could have hoped.”
Democrats could have made that happen a year earlier, when it might have been possible to actually remove Trump,
if they had run effective hearings about his corruption. Instead, Democratic leaders ran away. They fled. Why? They relied too much on polls, and didn’t believe they could use hearings to persuade people.
One other note. Yes, many Americans learned about Trump’s corruption due to Ukraine. Yes, that seems crazy to the highly-politically engaged – which means you, if you’re reading this article. But many Americans are too busy with their own lives to spend a lot of time reading about politics. Those Americans didn’t know, and still may not know, that Trump is corrupt and his corruption hurts them. This is why it would have been so valuable for Democrats to make the case to America, with repeated hearings and inherent contempt, that Trump is a corrupt criminal.
Persuasion requires bold action, informed by expertise – not decisions based on short-term polling data
This idea that one shouldn’t be tied to polls at all times is accepted outside of politics.
While digital marketing can also be very focused on short-term ROI measures, marketing professionals also use persuasion. Apple is famous for having run large branding campaigns to make people feel good about its products. If Apple marketing execs had to show polling data that those ads were creating more purchases immediately, in the short term, it seems unlikely that those campaigns would have ever gotten off the ground. What happened instead was that an expert in marketing took bold action, without strong data, and reshaped public opinion over time.
There are other domains that illustrate that being tied to data is limiting when it comes to complex systems. Sports statistics shows an example. In baseball, each pitch is mostly independent of each other pitch. That means one can collect data on large numbers of pitches, and use statistical models to deeply inform decisions about which pitchers are undervalued. But in American football, the action on each play is dependent on the action of many players. The quarterback might be throwing interceptions because he makes bad decisions – or his offensive line may be limited, his coach may use schemes unsuited to his skills, or he may be best at short passes but his receivers may be good at getting separation on long passes. That means you can’t rely on data exclusively to make decisions about football. If you tried to take a purely data-based approach, your decisions would be bad, and your team would lose. For this reason, football statistics have been limited for decision-making. Instead, the best coaches and teams are those who have skilled, experienced coaches who are able to implement plans based partly on intuition developed from long experience. Their decisions are informed by data, but ultimately the best decisions are made by experts who can take bold action, know when to ignore short-term data, and are executing a long-term plan.
To come: what we can do to get our Democratic leaders to stop over-relying on polls.
