On Why Those Polls Keep Misleading The Masses

The polls keep drastically confounding expectations, and lots of people want to know what's going on. How can something as relatively straightforward as asking people which way they will vote and collating statistics on that basis repeatedly turn out be so difficult? Like the weather, it should be impossible to nail it to an exact science, but surely the indicators should be more reliable than this, shouldn't they?

I don't want to claim to have a sixth sense - it's probably just good intuition, but the last three big political events have been cases where I felt confident of the result, even though the pollsters looked to convince me I was wrong. The majority seemed confident that Ed Miliband's Labour would win more seats than David Cameron's Conservatives last May, but I felt quite confident that the public wouldn't buy Miliband's message. Ditto Brexit - even though just before bedtime on the eve of the results I felt confident that we'd vote to leave the EU, the pollsters and bookmakers declared it highly unlikely. And now the same with Trump, who was a 5-1 outsider a few hours before the result, but who confounded majority expectations and won the Presidency. Assessing the feeling in the run-up, this wasn't much of a surprise to me either.

My record is not perfect, however - for quite a while I remained unconvinced that rank outsider Jeremy Corbyn would win the Labour leadership election the first time around against what I thought were more mainstream candidates (Andy Burnham, Liz Kendall & Yvette Cooper) - but I was completely wrong about the extent to which hoards of people were ready to vote for someone so far on the hard left.

Given that society can be divided into far more groups that many people ever do, there are clearly important sub-sections of society that are not getting captured in the polls. And anyway, populations are very large, and at best polls are only a small sampling of a demographic - it's difficult to take the pulse of the electorate if you place the measuring instrument on the sole of the nation's foot.

Polls are also seductive to people because many find it difficult to accept open-endedness and uncertainty in life. People are tendentiously drawn to simple stories with binary subplots that fit their preconceptions. The notion that things like elections and referendums are often beyond the scope of human predictions is something that few people wish to embrace, which is probably why the egoism of 'expert'-based prediction hangs around.

What's evident, aside from how polls fail to capture so much public opinion, is that polls are unreliable at capturing the following sets of society: the set in which people have been unsure how they were going to vote, or been unwilling to say, or been susceptible to an unexpected change of mind, or not been honest or open, and also the set in which people for whatever reason thought they'd vote one way but ended up not voting at all.

We have to face up to the fact that there is a huge epistemological gulf between the pre-voting opinion polls and what people have actually voted for on the day. The only sure-fire opinion poll of reliability is the result itself.

Finally, what would help people predict with greater accuracy would be an increased understanding of what the societal landscape is actually like. Taking the UK voting population as a whole, they can be divided roughly into 5 groups.

Group 1) Older people who think the nation needs a big state to govern its people, and like (within reason) the kind of governance we have at present.

Group 2) Younger people who think the nation needs a big state to govern its people, and like (within reason) the kind of governance we have at present.

Group 3) Older people who think the nation needs a big state to govern its people, and dislike the kind of governance we have at present.

Group 4) Younger people who think the nation needs a big state to govern its people, and dislike the kind of governance we have at present.

Group 5) People (older and younger) who think the nation does not need a big state to govern its people, and are gradually trying to influence change in a more libertarian framework of thinking.

Group 5 is currently the smallest group but is also quite a rapidly proliferating group. They are the people who would likely vote if economically savvy pro-market candidates came along, but may not be inclined to vote for the kind of politicians we have at the moment. Group 4 is a large group and is also a rapidly proliferating group. It consists of conscientious, left wing, green-friendly social justice warriors who want a big change of personnel in government. They want more of the same size government (bigger if possible), just not this kind of big government. This group, along with group 3, also feel quite stridently that inequality is a huge problem, that the government screws over the poor, and that the systems needs a complete socialist revolution type of overhaul.

Groups 1 and 2 are the two large groups (with group 1 being a lot larger than group 2) that help perpetuate the status quo by being somewhat complacently accepting of the system as it currently is, and not very much into revolutionary politics. These groups, in particular group 1, consist of lots of people who just vote one way out of habit, dyed in the wool allegiance, or just blithely out of a sense of duty to partake in democracy.

Trying to predict outcomes of elections and referendums involves getting a good grasp, not just of these 5 groups, but also of the multitude of potential sub-groups into which they could divide in the run up to voting day. That's why, once you offer the population a Remain or Leave vote in the EU Referendum, and then in the build up throw in all kinds of complications like reliability of claims, forecasting the cross-national outcomes of either decision, economic projections related to trade deals, immigration and market fluctuations, it becomes a very intractable business to predict - and ditto the UK General Election and the recent Presidential Election.