Judge online polling by real-world accuracy, not academic theory

Judge online polling by real-world accuracy, not academic theory

By Angus Reid

My commitment to the online methodology came from an epiphany of sorts that I had at the turn of the millennium, when non-participation rates for conventional polling were starting to top 90 per cent.  It became clear to me that we must find a better way to bring potential survey respondents into the research process – especially in a world that places more emphasis than ever on personal privacy and where robo-calls and even live interviewer engagement are seen as spam.

If enough care and attention (and dollars) are committed to this task, it is possible to build very large panels of “double opt-in” respondents – that is, people who choose to be part of the panel and then choose whether to participate in the surveys offered to them. With the proper investment, these panels can be large enough to represent each of the major regions and segments of a given country. Following on the footsteps of the very successful YouGov operation in the U.K., I started building panels in Canada, the U.S. and Britain.

The online panel offers some major advantages over other forms of research. Surveys can be completed on mobile devices at the respondent’s convenience and can include pictures and video to obtain a more realistic response context. Most of the investment in online research goes into maintaining and growing a large group of potential respondents – rather than paying for interviewers in call centres.

Because online polls involve sampling from a pre-recruited group willing to take a survey, some have attacked this method, claiming that the sample is not truly random and therefore the margin of error typically put out when a poll is released (e.g. “accurate +/- 4 %, 19 times out of 20”) cannot be used.

While this is technically correct about online polls, it is arguable that no poll today should use a margin of error, given the very serious problems of low completion- and high refusal-rates that inhibit a truly random sample.

Adding to the complexity and confusion is a lack of understanding on part of many reporters and editors who cite polls, and find themselves invariably skeptical of any poll that doesn’t report a margin of error.

In this environment, it is more appropriate to judge pollsters on their real-world performance than through the use of abstract mathematical models. We are living in a time of rapidly changing communication technology and, unfortunately, the standards used to assess polling are rooted in the wrong century.

In the polling world, there are two types of measures to assess the quality of election polling. The first, and most important, is picking the eventual winner. The second involves the level of accuracy surrounding the final projection. In golf parlance, it’s “how close did we get to the pin?”

On the first of the standards – picking the eventual winner – my accuracy has been 95 per cent. Ironically the two we missed were closest to my home in Vancouver: Alberta in 2012 and British Columbia in 2013. (Pollsters across all methods missed B.C. in 2013, suggesting something other than polling method problems were at work in that unusual election.)

In terms of precision, my average is better than three percentage points. In some cases, such as the 2011 federal contest and the 2012 U.S. Presidential election, we were off by one point or less. In other cases, we projected the winner but our margin of error was much higher (in Alberta in 2008, for example).

With the plethora of polling methods currently being deployed, it can be difficult to sort out results based on quality. Rather than leaving this determination to theoretical models, it makes more sense to judge the pollster by their record.

Below I have taken the liberty to reproduce my “election biography,” so readers can draw their own conclusions. I can’t speak for the entire polling industry, nor for others in the online field, but I’m quite happy with what we’ve been able to accomplish in eight years of working with a completely new research technology.

Our Electoral Record

Canadian National Elections

Canada – November 2015

In this contest, Angus Reid correctly projected the election with an average error of 1.7%. Notably, these data were pulled out of field 3 days before the election, whereas other firms polled until 1 or 2 days before.

Canada – May 2011

In this contest, Angus Reid squared off against eight other firms, including three which relied on online panels for some of their data collection. The final survey perfectly predicted the level of support that three of the five contending parties—Liberals, Bloc Québécois and Greens—would get at the ballot box.

Canada – October 2008

Angus Reid, using its online approach in a national election for the first time, conducted the only poll that foresaw a high number of votes for the Conservatives, a perfect prediction for the Green Party, and a double-digit difference between Tories and Grits at the national level.

Canadian Provincial Elections

Nova Scotia – October 2013

Angus Reid applied lessons learned from past Canadian elections and used weighting to mimic past voting patterns. The results correctly predicted a Liberal win.

Nova Scotia – June 2009

The Angus Reid surveys conducted during this provincial campaign anticipated a third place finish for the Progressive Conservatives, and a victory for the New Democrats—who had never formed the government in any Atlantic Canadian province.

Quebec – April 2014

Using the likely voter model first developed for the 2013 Nova Scotia election, we weighted respondents by those who are most likely to actually vote. Again the weighting formula worked as our results were within the margin of error for all four major parties.

Quebec – December 2008

The December 2008 election in Quebec provided a chance to review the effect of two emerging parties in the provincial landscape. The final Angus Reid survey anticipated that the governing Liberals would receive 42 per cent of the vote, while other firms that relied on the telephone overstated their predictions.

Quebec – March 2007

This was the first prediction of a Canadian democratic process that relied entirely on data collected from online research. While three other firms anticipated a third place finish for the opposition Action démocratique du Québec, the Angus Reid surveys saw a steady rise in support for the ADQ that would ultimately lead to the party becoming the official opposition in Quebec.

Ontario – June 2014

We adjusted the likely voter model first developed for the 2013 Nova Scotia election to take into account historic voting patterns in Ontario. However in this election our estimate based on all eligible voters and not just the likely voters was closer to the actual results, demonstrating that the likely voter model continues to be a work in progress.

Ontario – October 2011

Going into field on the final day of the campaign, our research noticed a shift towards the governing Liberal Party that materialized on election night.

Ontario – October 2007

The election was marked by the decision of the opposition Progressive Conservatives to call for specific changes to education guidelines. In the end, all participating parties were predicted within two percentage points of their actual result.

Manitoba – October 2011

The voting intention survey conducted days before the provincial ballot provided an exact prediction for three of the four contending parties – New Democrats, Liberals and Greens – and a difference of a single point for the Progressive Conservatives.

Manitoba – May 2007

An Angus Reid survey conducted a few days before the provincial election was held showed that incumbent premier Gary Doer was unquestionably more popular than opposition challenger Hugh McFadyen. The voting intention numbers confirmed that the governing NDP would win the election by a double-digit margin.

Saskatchewan – November 2007

A series of Angus Reid surveys showed that respondents in Saskatchewan were ready for a change of government, as the popularity of sitting premier Lorne Calvert waned and opposition contender Brad Wall connected with the population. The final Angus Reid survey accurately predicted that more than half of voters would support the Saskatchewan Party in the election.

Alberta – May 2015

Alberta – April 2012

A late-shift saw a severe drop in support for the Wildrose Party on the final days of the campaign, which translated in a victory for the incumbent Progressive Conservatives.

Alberta – March 2008

Four of the five parties in this election, which featured a particularly low turnout, were called within the margin of error, and all contenders were listed in the correct order.

 

BC Election – 2017

 

BC Transit Plebiscite – May 2015

 

British Columbia – May 2013

The 2013 general election in British Columbia saw the BC Liberals, led by Christy Clark, climb back from the depths of public disfavor to win a majority.

Most pollsters – including Angus Reid Global – predicted victory for the BC New Democratic Party and leader Adrian Dix. A thorough post-election review of ARG’s practices and methodology indicates our polling missed primarily because of low voter turnout among younger voters. Weighting the responses of voters under the age of 35 to their share of the voting electorate, as opposed to weighting them to their share of the overall BC population would have shown a much narrower three-point lead for the NDP in the final stretch of the campaign. This would have supported the dynamic of the Liberals picking up critical momentum.

BC Referendum – August 2011

The last survey conducted before registered voters in British Columbia received their ballots to take part in the mail-in referendum on the future of the harmonized sales tax (HST) pegged support for the Yes side at 56 per cent. When all the ballots were counted, the Yes side emerged victorious with 55 per cent of the vote.

 

British Columbia – May 2009

This provincial election featured the debut of Real Ballot—a revolutionary approach that showed respondents the candidates that were running in their respective constituencies. This approach allowed researchers to provide an exact prediction for two opposition parties: the NDP and the BC Conservatives.

For more on the Angus Reid polling record, including Federal and State elections in the United States, please click here

Tags assigned to this article:
elxn42

No comments

Write a comment
No Comments Yet! You can be first to comment this post!

Only registered users can comment.

Sign up for the Angus Reid Institute Mailing List

Receive our latest releases first