The Pollsters Are Missing A Big Trick In Predicting Elections

I was looking at some of the poll responses from members of the public this morning on the news, and thought to myself, the way pollsters poll seems to me to be not the most effective way to gauge the likelihood of a party's share of the vote.

Think about it, the pollsters ask members of the public who they are going to vote for, which is a very limited question because the only information they distil from that question is how that person will vote.

Now, a better question to ask members of the public would be: Which party do you think is going to win the election?

Here's why. When you ask someone who they think will be win the election, their brain will rapidly run a gamut of recall in their mind's library of experiential protocols - sifting through all their recent political conversations, family voting intentions, Facebook posts from friends revealing their party preferences, etc, and draw on a kind of weighted average of those experiences, thereby being able to intuit who they think the likely winning party will be.

That is to say, in asking someone Which party do you think is going to win the election? - as opposed to Who will you vote for? - the pollsters are, in essence, capturing a proxy poll of a much wider part of the demographic, because they are extrapolating from the person's wider social circle. Asking someone Which party do you think is going to win the election? is a little bit like asking them about the voting intentions of their social circle - it's a kind of larger poll trapped inside the body of a smaller poll.

In a sense, the phenomenon I'm talking about is a development of what's known as The Wisdom of Crowds, which is where the average guess of a large selection of guesses usually turns out to be astonishingly close to the correct answer.

This Wisdom of Crowds phenomenon famously came about through scientist Francis Galton (he of eugenics infamy) when he conducted an impromptu experiment at a farmyard exhibition, whereby people were asked to guess the weight of an ox, with a prize being awarded to the person who made the best guess. There were just under 800 guesses, with the average guess being within 1% of the correct answer 1,197 pounds, beating not only most of the individual guesses but also those of alleged cattle experts. This is what is meant by The Wisdom of Crowds.

To qualify that, three things: firstly, it is important that the group members give their answers independently without being influenced by each other; secondly, this works much better if the group has diversity - the more diverse the better; and thirdly, this also works better when there is a correct answer to the question that's easy to ascertain.

Given the foregoing, why, then, should the average of a large selection of guesses by non-experts consistently be more accurate than more educated guesses by individual experts? I think the probable explanation is that when an individual makes a guess or posits an answer, he or she is cluttered with quite a bit of background distraction from the variety of other thoughts, feelings and sensations, and that taking the average over a huge variety of responses may go some way to cancelling out this effect.

In November last year I wrote a Blog explaining why polls keep misleading the masses, and I touched on something that I think is heart of what I'm on about today. I talked about what I felt was good intuition regarding the outcome of the last three big political events (UK Election in 2015, EU Referendum and US Presidential Election). I felt confident about the results quite simply because everywhere I went - be it at work, in pubs, on the Internet - the sense was that there were more people trusting Cameron than Ed Miliband, more people wanting to leave the EU than remain in it, and enough people trusting Trump and/or disking Hillary Clinton to see Trump become President.

Similarly, I think polls that gauged this type of intuition by asking Who do you think is going to win? are the way forward if you want to achieve a much more accurate reading of the electorate's pulse on these matters.