Policing the Pollsters – urgent priority or immaterial pursuit?

by Angus Reid | August 1, 2018 5:49 pm

By Angus Reid[1], Chairman

In late July the Canadian Market Research Industry Association (MRIA) announced that it was suspending operations due to funding problems and pending insolvency. I was a speaker and sponsor at a recent MRIA event in Vancouver and can attest to the lack of wider industry involvement there.

There are several thousand people working in the market research industry in Canada. Most are engaged in conducting and analysing surveys of consumers and citizens for private and public sector clients. This work is generally conducted in secret for clients who know a lot about market research and choose their consultants and suppliers very carefully. This private sector research usually stays private, while research conducted for government is usually released publicly, though the convention around this varies widely across Canada.

A small amount of the research conducted in Canada is carried out for public consumption. This includes surveys for the media and studies for corporations and special interest groups. The latter two are often looking for facts about public opinion to support their cause, image or story. This public polling lies at the heart of the challenges facing a trade organization like the MRIA. I follow this especially closely because all of the research we do at Angus Reid Institute is released publicly.  We are a charitable foundation under the purview of the Canada Revenue Agency and are required to release all of the work we do at no charge.

Public polling is the sharp end of the spear for the research industry everywhere. When the pollsters miss (or are perceived as missing) important events such as the Brexit referendum or the Trump election, then the work of the entire industry is drawn into question.

When MRIA announced it was shutting down, Canadian Press put out a story that “Canadian Pollsters are losing their watchdog”. Industry commentators claimed that a new association must be formed quickly to serve as a “standard bearer for quality research”. Such reportage assumes that there ever was a watchdog and that there is agreement among experts on quality research. Neither is the case.

Some of my colleagues refer to me as the grandfather of Canadian polling. I don’t feel that old but have been around long enough to see how changing technology challenges the warm feeling of comradery among the major players. When telephone polling started to unseat door-to-door as the method of choice back in the 1970s the industry began to fracture like sheet ice on a cold artic night. These transformations facilitate the entry of new players and upset many of the cozy relationships among competitors.

Over the last twenty years new online and interactive voice technology (IVR, or ‘Robo calling’) have transformed the market research industry into but a shadow of its former self. Telephone polling, once the mainstay of the industry has been largely pushed to the sidelines because of a combination of very low cooperation rates, cell phone use and cost. Today pollsters use such a wide variety of methods to measure the mood of the public that it is virtually impossible to agree on quality standards much less police them.

To be sure, there are massive quality differences. In Britain, several online pollsters like YouGov were far more accurate in predicting the outcome of the Brexit vote compared to others who missed the trend completely. In Canada the recent mayoralty race in Calgary showed just how wrong some pollsters can be – especially when they rely on IVR.

There are many good reasons for establishing a new organization to foster professional development and share best practices among seller and buyers of research services. But policing quality should be near the bottom of the list. Individual companies may wish to band together as co-ops to bless each other and their practices, but research buyers and the media shouldn’t be duped by this exercise when making judgments about research companies. The massive quality differences in the research arena require each organization to be judged on its own merits.

In the area of online research, the Washington based Pew Research Center recently carried out a quality assessment of a dozen companies in this space and found that most failed to meet even elementary standards. But a couple performed brilliantly.

Other countries – notably the US and UK have organizations whose sole purpose is to educate the public, politicians and the media about how to judge data on opinions that have been publicly released. Their role is less about policing and more about helping users draw conclusions from the many polls that now appear during most elections. In both cases they have conducted special investigations where pollsters missed the mark completely.

In Canada we don’t have the luxury of a more specialized association dealing solely with “public” public opinion polls. We’ve mashed everyone together – including the largest group who never conduct public polls. This breeds misunderstanding – and a harmful image for the industry as a whole – especially either when a couple of bad actors get it wrong – or as happens seldom – (BC, 2013) when everyone gets it wrong.

Like the purveyors of cheese, wine and dog food, the array of offerings in the research world vary widely in cost and quality. It’s important to have a strong association to share ideas and maybe sample each other’s fare but don’t confuse membership as evidence of high professional standing. These trade associations exist to serve the interests of their members.

Caveat Emptor.

  1. Angus Reid: http://www.angusreid.org/our-team/angus-reid

Source URL: https://angusreid.org/policing-the-pollsters-urgent-priority-or-who-cares/