For anyone watching what’s going on in the world, it’s been one hell of a week –and probably the longest we’ve had to wait for election results in the US in a while. Indeed, this year has been unique in so many aspects that it was expected that the general election in the US was going to fall within that pattern of a crazy 2020. And as we’re still seeing numbers emerge from a slow, almost nerve-racking vote count it is also worth looking at the weeks leading up to the election and the fact that pollsters got it wrong –yet again. 

The polls were pointing to a Democratic sweep which didn’t entirely happen as expected in, say, Florida or Wisconsin (which was predicted to be 6.7 points ahead for the Democratic Party).There’s also the argument among many, that polls are snapshots not predictions. What they do, is they give us an understanding of where things stand right now, but don’t necessarily indicate where they will stand in a week from now. Any way you look at it, from 2016 onwards, the weight, clarity, validity and importance of polls has not been what it was in the past. Interestingly enough, polling errors seem to favour liberals and fail conservatives.

One factor that skewed polls this year may have been the record turnout. According to the US Elections Project, the turnout was almost 67% of eligible voters –the highest rate since the 1900s. So, out of that percentage, which voters turned out that didn’t do so in the past? And most importantly, who is being left out of polls and why? Is there a specific group that is systematically excluded from polls? Let’s not forget that, interestingly enough, polling errors seem to favour liberals and fail conservatives. In the past, pollsters have come up with a voter profile called the “shy Trump voter”. This is a voter who does not disclose their preference for Donald Trump. What is becoming more apparent now, however, is that rather than being shy, Trump voters are simply not interested in responding to polls. The blame for that falls to the candidate himself. Trump has attacked polls –and the news—repeatedly in the past, so much so, that his voters are considering these polls as part of the “system” and Trump is their anti-systemic saviour.

At the same time, landlines are also becoming a rarity these days, which means that pollsters need, and have been applying new methods for their predictions. These include online surveys and text messages. However, finding the right ratio of phone calls vs online surveys and also accounting for the digital divide in the United States, which is getting bigger and harder to bridge as long as there is no funding to motivate ISPs to provide services to non-densely populated areas, can take long and prove to be quite tricky. 

The US is the most recent example, and one that created a lot of noise in the media, however we have seen this before and it appears to be a new pattern, especially in elections and referenda that result in (and feed on) highly polarised public opinion. Pollsters got the UK referendum on Brexit wrong, and before that many got the Greek referendum wrong as well. So, where does innovation lie when it comes to polling and elections?

We have been talking about machine learning, data mining and AI in business for so long, yet when it comes to democratic processes, we often fail to understand how valuable these tools could prove to be. Besides regulation, one thing we could have learned by Cambridge Analytica, is how important data is in determining voter profiles, behaviours, preferences and problems. Surely, there are ways in which pollsters can get data to capture all that, much like tech giants use data to determine what you’re likely to buy next, what you might be interested in buying in the future and how your consumer behaviour is evolving.  Admittedly, data on behaviours might be harder to get, but not impossible, and at the same time, if we are looking at that direction there are also regulatory issues that will need to be addressed to make sure that such data is mined ethically, ensuring transparency and understanding on the public’s side.

We have recently seen a few projects taking a shot at this, some successfully others not so much. One interesting example is Expert.ai, an Italian software company, analysed posts about Trump and Biden –millions of them. The AI system was in part trained on past elections and was able to analyse the emotion or tone of these posts and then project that in votes. The system predicted a 50.2% win for the Democrats and 47.3% for Republicans. On Friday, percentages were 50.5% and 47.8% respectively. Polls are indeed important. Not just for predicting elections. Not for politicking. They are important in creating a dialogue between leaders and voters. They help democracies stay healthy (or as healthy as possible), by allowing leaders to understand what the people are thinking. Bill Clinton was known for his use of polls to guide his decisions and perhaps lucky because during his time polls were working. What happens in the era of Twitter and online campaigns?

Projects like Expert.ai are most likely heading to the right direction as past methods now seem obsolete. There is an undeniable need for change and evolution if we are determined to push our democracies further and transform them according to the needs of this new era we’re entering. And perhaps AI could provide a lot of the answers we are seeking by extending past data to predict the direction of future change. Besides, it has been doing the very same thing for your online purchases.