Judge online polling by real-world accuracy, not academic theory

September 1, 2015 – By Angus Reid

I’ve covered almost 40 elections over the past eight years – all of them using online panels that I built in Canada, the U.S. and the U.K. while I was CEO of Vision Critical. In that time, our record has demonstrated that online polling can be accurate and reliable.

My commitment to the online methodology came from an epiphany of sorts that I had at the turn of the millennium, when non-participation rates for conventional polling were starting to top 90 per cent.  It became clear to me that we must find a better way to bring potential survey respondents into the research process – especially in a world that places more emphasis than ever on personal privacy and where robo-calls and even live interviewer engagement are seen as spam.

If enough care and attention (and dollars) are committed to this task, it is possible to build very large panels of “double opt-in” respondents – that is, people who choose to be part of the panel and then choose whether to participate in the surveys offered to them. With the proper investment, these panels can be large enough to represent each of the major regions and segments of a given country. Following on the footsteps of the very successful YouGov operation in the U.K., I started building panels in Canada, the U.S. and Britain.

The online panel offers some major advantages over other forms of research. Surveys can be completed on mobile devices at the respondent’s convenience and can include pictures and video to obtain a more realistic response context. Most of the investment in online research goes into maintaining and growing a large group of potential respondents – rather than paying for interviewers in call centres.

Because online polls involve sampling from a pre-recruited group willing to take a survey, some have attacked this method, claiming that the sample is not truly random and therefore the margin of error typically put out when a poll is released (e.g. “accurate +/- 4 %, 19 times out of 20”) cannot be used.

While this is technically correct about online polls, it is arguable that no poll today should use a margin of error, given the very serious problems of low completion- and high refusal-rates that inhibit a truly random sample.

Adding to the complexity and confusion is a lack of understanding on part of many reporters and editors who cite polls, and find themselves invariably skeptical of any poll that doesn’t report a margin of error.

In this environment, it is more appropriate to judge pollsters on their real-world performance than through the use of abstract mathematical models. We are living in a time of rapidly changing communication technology and, unfortunately, the standards used to assess polling are rooted in the wrong century.

In the polling world, there are two types of measures to assess the quality of election polling. The first, and most important, is picking the eventual winner. The second involves the level of accuracy surrounding the final projection. In golf parlance, it’s “how close did we get to the pin?”

On the first of the standards – picking the eventual winner – my accuracy has been 95 per cent. Ironically the two we missed were closest to my home in Vancouver: Alberta in 2012 and British Columbia in 2013. (Pollsters across all methods missed B.C. in 2013, suggesting something other than polling method problems were at work in that unusual election.)

In terms of precision, my average is better than three percentage points. In some cases, such as the 2011 federal contest and the 2012 U.S. Presidential election, we were off by one point or less. In other cases, we projected the winner but our margin of error was much higher (in Alberta in 2008, for example).

With the plethora of polling methods currently being deployed, it can be difficult to sort out results based on quality. Rather than leaving this determination to theoretical models, it makes more sense to judge the pollster by their record.

Below I have taken the liberty to reproduce my “election biography,” so readers can draw their own conclusions. I can’t speak for the entire polling industry, nor for others in the online field, but I’m quite happy with what we’ve been able to accomplish in eight years of working with a completely new research technology.

Our Electoral Record

Canadian National Elections

Canada – November 2015

In this contest, Angus Reid correctly projected the election with an average error of 1.7%. Notably, these data were pulled out of field 3 days before the election, whereas other firms polled until 1 or 2 days before.

Actual Angus Reid Ipsos Nanos Leger Forum Ekos
Lib. 39.50% 35% 38% 38% 38% 40% 36%
Con. 31.90% 31% 31% 30% 30% 30% 32%
NDP 19.70% 22% 22% 21% 22% 20% 20%
BQ 4.70% 5% 4% 5% 6% 6% 5%
Grn. 3.50% 5% 4% 5% 4% 3% 6%
Av. Err. -1.7% 1.2% 1.3% 1.5% 0.9% 1.3%

Canada – May 2011

In this contest, Angus Reid squared off against eight other firms, including three which relied on online panels for some of their data collection. The final survey perfectly predicted the level of support that three of the five contending parties—Liberals, Bloc Québécois and Greens—would get at the ballot box.

Actual Angus Reid Ipsos Nanos Leger Forum Decima Abacus Ekos Compas
Con. 40% 37% 38% 37% 36% 36% 36% 37% 34% 46%
NDP 31% 33% 33% 32% 31% 33% 30% 32% 31% 26%
Lib. 19% 19% 18% 21% 21% 19% 19% 18% 21% 17%
BQ 6% 6% 7% 6% 7% 6% 6% 7% 6% 7%
Grn. 4% 4% 4% 4% 4% 5% 6% 7% 6% 4%
Av. Err. -1.0 -1.2 -1.2 -1.4 -1.4 -1.4 -1.8 -2.0 -2.8

Canada – October 2008

Angus Reid, using its online approach in a national election for the first time, conducted the only poll that foresaw a high number of votes for the Conservatives, a perfect prediction for the Green Party, and a double-digit difference between Tories and Grits at the national level.

Actual Angus Reid Ekos Leger Ipsos Decima Nanos TSC Segma
Con. 38% 37% 35% 36% 34% 34% 34% 33% 35%
Lib. 26% 27% 26% 27% 29% 25% 27% 28% 23%
NDP 18% 20% 19% 20% 18% 19% 21% 18% 21%
BQ 10% 9% 10% 8% 9% 11% 10% 10% 10%
Grn. 7% 7% 10% 9% 8% 9% 8% 11% 11%
Av. Err. -1.0 -1.4 -1.8 -1.8 -1.8 -1.8 -2.2 -2.6
Canadian Provincial Elections

Nova Scotia – October 2013

Angus Reid applied lessons learned from past Canadian elections and used weighting to mimic past voting patterns. The results correctly predicted a Liberal win.

Actual Angus Reid CRA
New Democratic Party (NDP) 27% 27% 29%
Liberal Party 46% 46% 52%
Progressive Conservative Party 26% 24% 17%
Other 1% 3% 2%
Av. Err. -1.0 -4.5

Nova Scotia – June 2009

The Angus Reid surveys conducted during this provincial campaign anticipated a third place finish for the Progressive Conservatives, and a victory for the New Democrats—who had never formed the government in any Atlantic Canadian province.

Actual Angus Reid CRA
New Democratic Party (NDP) 45% 47% 44%
Liberal Party 27% 26% 28%
Progressive Conservative Party 24% 23% 26%
Green Party 2% 3% 2%
Other 1% 1% 0%
Av. Err. -1.0 -1.0

Quebec – April 2014

Using the likely voter model first developed for the 2013 Nova Scotia election, we weighted respondents by those who are most likely to actually vote. Again the weighting formula worked as our results were within the margin of error for all four major parties.

Actual Angus Reid Forum Leger Ekos
Liberal Party (PLQ) 41.5% 39% 44% 38% 40%
Parti Québécois (PQ) 25.4% 27% 24% 29% 27%
Coalition Avenir Québec (CAQ) 23.1% 25% 23% 23% 21%
Québec Solidaire 7.6% 7% 6% 9% 9%
 Av. Err. -1.7 -1.4 -2.2 -2.6

Quebec – December 2008

The December 2008 election in Quebec provided a chance to review the effect of two emerging parties in the provincial landscape. The final Angus Reid survey anticipated that the governing Liberals would receive 42 per cent of the vote, while other firms that relied on the telephone overstated their predictions.

Actual Angus Reid Leger CROP Nanos
Liberal Party (PLQ) 42% 42% 45% 45% 44%
Parti Québécois (PQ) 35% 36% 32% 29% 36%
Action démocratique du Québec (ADQ) 16% 13% 15% 15% 12%
Québec Solidaire 4% 5% 4% 5% 5%
Green Party 2% 3% 3% 6% 5%
 Av. Err. -1.2 -1.3 -3.2 -2.2

Quebec – March 2007

This was the first prediction of a Canadian democratic process that relied entirely on data collected from online research. While three other firms anticipated a third place finish for the opposition Action démocratique du Québec, the Angus Reid surveys saw a steady rise in support for the ADQ that would ultimately lead to the party becoming the official opposition in Quebec.

Actual Angus Reid Leger CROP TSC
Liberal Party (PLQ) 33% 31% 35% 34% 30%
Action démocratique du Québec (ADQ) 31% 30% 26% 25% 28%
Parti Québécois (PQ) 28% 29% 29% 28% 31%
Av. Err. -1.3 -2.7 -2.3 -3

Ontario – June 2014

We adjusted the likely voter model first developed for the 2013 Nova Scotia election to take into account historic voting patterns in Ontario. However in this election our estimate based on all eligible voters and not just the likely voters was closer to the actual results, demonstrating that the likely voter model continues to be a work in progress.

Actual Angus Reid (eligible) Angus Reid (likely) Abacus Data (eligible) Abacus  Data (likely) Ipsos-Reid (eligible) Ipsos-Reid (likely)
Liberal Party 38.7% 36% 34% 35% 36% 33% 30%
Progressive Conservative 31.3% 32% 36% 32% 36% 33% 36%
NDP 23.8% 26% 24% 26% 23% 30% 30%
Green 4.8% 5% 5% 6% 5% 4% 3%
 Av. Err. -1.5 -2.5 -2.0 -2.1 -3.6 -5.4

Ontario – October 2011

Going into field on the final day of the campaign, our research noticed a shift towards the governing Liberal Party that materialized on election night.

Actual Angus Reid Forum Abacus Nanos Ekos Ipsos
Liberal 38% 37% 37% 37% 38% 38% 41%
Prog Con. 35% 33% 36% 34% 33% 32% 31%
NDP 23% 26% 23% 24% 26% 23% 25%
Green 3% 3% 3% 4% 2% 6% 3%
Av. Err. -1.2 -0.4 -0.8 -1.2 -1.4 -1.8

Ontario – October 2007

The election was marked by the decision of the opposition Progressive Conservatives to call for specific changes to education guidelines. In the end, all participating parties were predicted within two percentage points of their actual result.

Actual Angus Reid SES Enviro. Ipsos TSC Decima
Liberal Party 42% 40% 43% 46% 43% 42% 42%
PC Party 32% 34% 31% 31% 32% 27% 31%
NDP 17% 19% 18% 20% 18% 19% 17%
Green Party 8% 7% 9% 3% 6% 11% 10%
Av. Err. -1.8 -1.0 -3.3 -1.0 -2.5 -0.8

Manitoba – October 2011

The voting intention survey conducted days before the provincial ballot provided an exact prediction for three of the four contending parties – New Democrats, Liberals and Greens – and a difference of a single point for the Progressive Conservatives.

Actual Angus Reid Probe Environics

Viewpoints

New Democratic Party (NDP)

46% 46% 46% 42% 51%
Progressive Conservatives 44% 43% 43% 45%

39%

Liberal Party

8% 8% 7% 10% 6%
Green Party 3% 3% 4% 3%

4%

Manitoba – May 2007

An Angus Reid survey conducted a few days before the provincial election was held showed that incumbent premier Gary Doer was unquestionably more popular than opposition challenger Hugh McFadyen. The voting intention numbers confirmed that the governing NDP would win the election by a double-digit margin.

Actual Angus Reid Probe
New Democratic Party (NDP) 48% 49% 44%
Progressive Conservative Party 38% 37% 37%
Other parties 14% 15% 19%
Av. Err. -1.0 -3.3

Saskatchewan – November 2007

A series of Angus Reid surveys showed that respondents in Saskatchewan were ready for a change of government, as the popularity of sitting premier Lorne Calvert waned and opposition contender Brad Wall connected with the population. The final Angus Reid survey accurately predicted that more than half of voters would support the Saskatchewan Party in the election.

Actual Angus Reid Sigma
Saskatchewan Party 52% 52% 54%
New Democratic Party (NDP) 37% 35% 34%
Liberal Party 10% 12% 8%
Av. Err. -1.3 -2.3

Alberta – May 2015

Actual Angus Reid Forum Leger Insights West EKOS
NDP 41% 41% 45% 38% 42% 44.3%
PC 28% 28% 23% 30% 27% 24%
Wildrose 24% 22% 23% 24% 23% 22.5%
Liberal 4% 5% 4% 6% 4% 5.6%
Other 3% 5% 5% 2% 5% 3.6%
Av. Err. -1.0 -2.4 -1.6 -1.0 -2.2

Alberta – April 2012

A late-shift saw a severe drop in support for the Wildrose Party on the final days of the campaign, which translated in a victory for the incumbent Progressive Conservatives.

Actual Angus Reid Forum Leger CR Abacus
PC 44% 32% 36% 36% 34% 31%
Wildrose 34% 41% 38% 42% 41% 41%
Liberal 10% 13% 10% 9% 11% 12%
NDP 10% 11% 12% 10% 11% 13%
Other 2% 2% 4% 3% 3% 3%
Av. Err. -4.6 -3.2 -3.6 -4.0 -5.2

Alberta – March 2008

Four of the five parties in this election, which featured a particularly low turnout, were called within the margin of error, and all contenders were listed in the correct order.

Actual Angus Reid TSC Leger
Progressive Conservatives 53% 43% 50% 55%
Liberal Party 26% 28% 25% 24%
New Democratic Party 9% 13% 8% 7%
Wildrose Alliance Party 7% 10% 10% 8%
Green Party 5% 7% 8% 6%
Av. Err. -4.2 -3.2 -2.2

BC Transit Plebiscite – May 2015

Actual Angus Reid
Yes 38%

41%

No

62% 59%
Difference

-3

British Columbia – May 2013

The 2013 general election in British Columbia saw the BC Liberals, led by Christy Clark, climb back from the depths of public disfavor to win a majority.

Most pollsters – including Angus Reid Global – predicted victory for the BC New Democratic Party and leader Adrian Dix. A thorough post-election review of ARG’s practices and methodology indicates our polling missed primarily because of low voter turnout among younger voters. Weighting the responses of voters under the age of 35 to their share of the voting electorate, as opposed to weighting them to their share of the overall BC population would have shown a much narrower three-point lead for the NDP in the final stretch of the campaign. This would have supported the dynamic of the Liberals picking up critical momentum.

Actual Angus Reid Forum Justason Ipsos
Liberal 44% 36% 41% 31% 36%
NDP 40% 45% 43% 45% 43%
Green 8% 9% 8% 14% 10%
Conservative 5% 7% 6% 8% 7%
Other 3% 4% 2% 3% 3%
Av. Err. -3.4 -1.6 -5.4 -3.0

BC Referendum – August 2011

The last survey conducted before registered voters in British Columbia received their ballots to take part in the mail-in referendum on the future of the harmonized sales tax (HST) pegged support for the Yes side at 56 per cent. When all the ballots were counted, the Yes side emerged victorious with 55 per cent of the vote.

Actual

Angus Reid

Yes

55% 56%
No 45%

44%

Difference

-2

British Columbia – May 2009

This provincial election featured the debut of Real Ballot—a revolutionary approach that showed respondents the candidates that were running in their respective constituencies. This approach allowed researchers to provide an exact prediction for two opposition parties: the NDP and the BC Conservatives.

Actual Angus Reid Ipsos IRG Mustel Enviro.
BC Liberal Party 46% 44% 47% 46% 47% 47%
New Democratic Party (NDP) 42% 42% 39% 37% 38% 36%
Green Party 8% 10% 10% 11% 12% 12%
Other 4% 4% 4% 6% 3% 5%
Av. Err. -1.0 -1.5 -2.5 -2.5 -3.0
For more on the Angus Reid polling record, including Federal and State elections in the United States, please click here

Related Posts

medicare poll canada
Canada’s health care system has serious problems, so why isn’t this election about them?
Angus Reid Institute
Will uncertain voters follow their heads or their hearts?
On foreign policy, trade trumps, but can any party turn the TPP into votes?

Leave a Reply

Sign up for the Angus Reid Institute Mailing List

Receive our latest releases first