On Monday night, I watched with utter shock and embarrassment as results came in for municipal elections across Alberta and our final prediction in the Calgary municipal election was completely and totally wrong.
Our final tally showed an 11 point win for challenger Bill Smith over incumbent Naheed Nenshi. The result was a 7 point win for Nenshi. Our final poll had underestimated the incumbent’s vote by 12% and overestimated the challenger by 8% for a total deviation of over 20%. These results will be a constant reminder for me and my staff going forward that even a good sample can be wrong and we must constantly analyze our frame design. The details below provide some context about our polling in the Calgary municipal election and more about our methodology.
Before that, I want to say that I am sorry for this error and that I have and will continue to be accessible and accountable to the voters of Calgary for this unexplained error. Our Smart IVR platform has consistently and accurately performed well versus ballot results, not two or three times, but hundreds and hundreds of times across Canada, across Alberta & even in two recent by-elections in Calgary.
As a result of this error, I have made three immediate decisions.
- I have appointed the head of our Analytics Team, Dr. Joseph Angolano to conduct a root and branch review of the Calgary polling and our research methods in general. As a part of this review, he will be looking at every Mainstreet survey conducted over the last 24 months and has been given full latitude to complete this review, including engaging with academics and other experts in public opinion research. Results of the review will be posted to our website when Dr. Angolano has completed his review. I have not imposed any timeline on his work.
- I have moved all of our Alberta polling to our live call platform until recommendations from Dr. Angolano’s review can be implemented. Our live call sampling is used in Quebec, Ontario, and some of our national polling and in the US where IVR cannot be used to dial cellular phones.
- I have appointed David Valentin, Executive Vice-President of Mainstreet, to lead an internal technological assessment that will review both our Chimera and Smart IVR systems.
Alberta Municipal Polling
Starting in late 2014, we began polling for the Postmedia network and all its newspapers across Canada including the Calgary Herald & Edmonton Journal in Alberta. With the addition of the QMI chain, Postmedia also owns the Edmonton & Calgary Sun. Our first polls focused on provincial politics, and monthly Calgary & Edmonton polling content was added in early 2015.
Our polling in the Alberta municipal election cycle began in late September, a poll of 1000 Calgarians was conducted via IVR on September 28th, 2017 that showed a 9 point lead for Bill Smith. Subsequent polls showed a 17 point lead on Oct 3rd & 4th and finally a 13 point lead in our final poll on Oct 10th & 11th. The first two polls were highly criticized for what was admittedly odd results, including a Smith lead among women and those under 35 which did not match past support or typical partisan voter behaviour.
The criticism of our first poll results and methodology did prompt some review of our work internally and led to a change in the second poll. The criticism included sample size and geographic distribution of the sample.
The first sample was conducted citywide using random selection, but even random selection can inadvertently sample more from certain parts of the City than others that may favour one candidate over another. Important to note that we did review the first sample for geographic distribution and it was not oversampling from a specific ward or region.
The sample size of 1000 for a population the size of Calgary was sufficient, but we agreed to increase it to 1500 to address this. We adjusted the frame design to a ward model, meaning instead of randomly selecting 1000 people from across Calgary, we randomly selected approximately 100 from each of the 14 Calgary wards. This ward model addressed the concerns but led to amplification of what I believe was ultimately the problem in our frame (more on that below).
Testing the Master Sample
Given the level and seriousness of the criticism of the first poll, we tested the second sample against five ward oversamples. The 1500 responses in the citywide sample had responses approximately equal to each other in each of the fourteen wards. Those results were tested against 600 samples taken in five randomly selected wards to determine if the ward sub-sample of 100 was consistent with the ward oversample of 600. In each of the five wards we tested, the results were consistent. Based on that, we were very confident in the master sample that showed a 17 point lead for Smith among all voters. It is important to note that not only was the master sample was tested overall to the wards, and each demographic subset was examined for consistency. In each case, men, women, youth, seniors and every group in the ward samples & city-wide sample was behaving consistently.
Mode Effect
Beginning in the second week of our polling, we added an online component to test for mode effect. Mode effect is measured by the differences in responses to questions using different modes. If we ask people the same question online, over the phone using live callers or on the phone using IVR, we usually see different results, this is measurable and was consistently showing a Nenshi bump among online respondents. We know that the Nenshi campaign relied on online surveys to track their internal numbers, so the criticism was expected as our online sampling showed a 4% lead for Nenshi, including a 30% lead among those 18-34.
In the 2015 Alberta election, the mode effect was so pronounced that our online survey showed greater than 50% support for the Alberta NDP in the closing days of the campaign. Our final poll report, using IVR, fielded seven days before the election had it pegged at 44% but samples taken closer to election day showed it pulling back closer to 42% with the incumbent PCs surging to 25% while online panels showed the reverse, a growth in NDP support. The final result was closer to 40% and that the Alberta PCs grew to 26%.
Blending with Online
In 2016, Mainstreet Research asked the Fields Institute at the University of Toronto to look at the possibility of “blending” online and IVR samples to solve the growing problem of lower response rates among cell phones and landlines. This study and results can be found here. It’s important to note that the accuracy of the online samples was largely dismissed by the IPSW participants when measured against actual results and that blending of online samples with IVR increased the error, it never reduced it. Sampling done in Manitoba, Saskatchewan & Alberta provincial elections and the last federal election were used for this review.
As a matter of full disclosure, part of the online samples that were looked at during this study were ones provided from those taken by Brian Singh during the Alberta election. Brian Singh conducted the internal polling for the Naheed Nenshi campaign.
As an experiment, I asked my staff to blend an IVR sample taken during the second week of October with the online sample of the same timeframe and to provide ward weighted results and citywide weighted results for comparison. It yielded the following results.
Online responses blended with IVR weighted by gender, age, and ward:
| All | Male | Female | 18-34 | 35-49 | 50-64 | 65+ | |
| Andre Chabot | 5% | 4% | 7% | 7% | 5% | 4% | 5% |
| Bill Smith | 42% | 46% | 37% | 25% | 41% | 57% | 55% |
| Naheed Nenshi | 38% | 37% | 39% | 41% | 43% | 30% | 32% |
| Someone Else | 3% | 3% | 3% | 6% | 2% | 1% | 1% |
| Undecided | 12% | 10% | 14% | 20% | 10% | 7% | 7% |
Online responses blended with IVR weighted by gender and age only (ward excluded):
| All | Male | Female | 18-34 | 35-49 | 50-64 | 65+ | |
| Andre Chabot | 5% | 4% | 6% | 7% | 4% | 5% | 5% |
| Bill Smith | 40% | 44% | 35% | 22% | 40% | 56% | 52% |
| Naheed Nenshi | 39% | 37% | 40% | 42% | 46% | 30% | 33% |
| Someone Else | 3% | 3% | 3% | 7% | 2% | 1% | 1% |
| Undecided | 13% | 11% | 15% | 22% | 8% | 8% | 9% |
A few things to note immediately as follows.
- The ward weighting amplifies the relative lead held by Bill Smith.
- “Blending” with online responses still yielded a Smith lead in each case, albeit a much smaller one
Now the obvious question that will no doubt be asked is the following.
Why did Mainstreet Research not publish these results instead of the IVR only results?
The answer is simple and evidence-based. Results of the Fields Institute Industry Problem Solving Workshop show strong evidence that “blending” of online with IVR results would produce results that contained more error, not less when measured against actual results in each of the four elections they examined. Our confidence in our published final numbers was not based on bravado and ego as some have suggested, it was based on solid evidence and a long track record of the accuracy of SmartIVR over other modes.
The less obvious suggestion might be made that things might have changed significantly since 2015/2016 elections and that may be a fair comment. Our most recent general election polling using IVR in Nova Scotia was accurate within the margin of error as was the IVR polling conducted by Forum Research. The most recent example of IVR’s accuracy came just weeks before the Calgary Municipal election in the by-election of Louis Hebert, that poll is found below.
Again, Smart IVR methodology proved correct in what is largely considered the hardest type of voter behaviour to predict accurately, namely by-elections. Turnout is volatile, and a small change can affect the accuracy. This continued accuracy of IVR also made us confident in the Calgary poll results.
Track Record & Alberta Results
As we pointed out during the course of the campaign, allegations made that our track record was ideologically driven or anything but stellar were disappointing. Our final published poll of the last Alberta election and prediction were within the margin of error. Our final sample just two days before the election, did show a late surge of the PCs to 25% and a pullback of the NDP support to 42%. Those numbers were shared at the time widely. When some pointed out that our final Alberta poll failed to reflect accurately final vote outcomes, it was disappointing that those who did receive the final update did not point that out. Regardless, our results were in line with most polls published, including online panels from Insights West & Think HQ that similarly underestimated final PC support. The suggestion that this was evidence of a methodological flaw was and is wrong, the evidence from past elections definitively proves it.
Although some critique over the poll was fair commentary, specifically the inexplicable shifts in 18-34 support in the second poll and the big difference in the way men and women behaved in the first poll, the basis for much of the critique from the academic pundits was and remains unfounded. If our results from the last Alberta Provincial election are evidence of a fundamental methodological flaw, then all the methodologies must be similarly fundamentally flawed because all modes, all methodologies, yielded similar results. That cherry-picking type of analysis was why we took exception with some of the criticism.
A few specific things were brought up throughout the campaign that I would like to address.
Our final poll in the Saskatoon Mayoral election in 2016 showed the incumbent Don Atcheson heading for a narrow win in what was a see-saw three-way battle. On election day, challenger Charlie Clark outperformed the poll by 7% on election day to beat the incumbent by 3 points.
Clark had been third the previous week and the late shift was caught by the IVR and continued in the last 48 hours that ultimately gave Clark the win.
It is important to note that the only other poll in the field was an online panel poll that projected the same win for Atchison, but had Charlie Clark at just 21% and in third place. Our online survey that was conducted at the time showed the exact same thing, so the panel poll results did not surprise us and gave us another measure of mode effect in civic elections.
Ultimately, I believe both polls did well in what was a heated three-way race in which voter patterns changed rapidly.
What is also very important to consider for context is that Mainstreet Research was the only published poll in the Regina mayoral election and all numbers were within the margin of error despite the field date being well ahead of the election. The reason this is important is because of the often cited examples of what is called “herding.”
In the second week of our polling, we began to see posts referencing the MRIA sanction of Mainstreet Research. Our statement on this matter can be found here. As it points out, a legal action has been launched so no further comments can be made by Mainstreet. However, our lawyer is happy to comment further.
A few things that I can say are the following,
- Mainstreet Research proactively disclosed the sanctions to all clients and sent all media across Canada both the sanction notice & our statement in April 2016.
- Our media partners at Postmedia and other chains were provided with the full adjudication decision & the appeal panel decision to provide important context to the posted decision.
- Mainstreet Research’s Statement of Claim is now a matter of public record
- Mainstreet Research comments on this matter during the campaign were not an attempt to suppress information, only an attempt to maintain the legitimacy of the legal process now underway
- Mainstreet, Postmedia & Local Influencers
It has been suggested throughout the campaign and since election day, that Mainstreet and/or Postmedia coordinated the polling to influence the campaign in some way. Specifically, hurting Mayor Nenshi or his campaign. It was suggested that we coordinated polling with the Bill Smith campaign, conducted push polling, and/or worked for the Calgary Flames organization. None of that could be further from the truth.
We sampled using the same methodology we have used to accurately predict hundreds of election outcomes across North America, including dozens of others across Alberta on Monday night and reported those numbers.
We were so confident in our final numbers (the best sample we got in the entire election) that we published a “polling scorecard” on the night before the election for people to easily measure our relative accuracy. While that now further compounds our embarrassment, it is just a small indication of our honest confidence in the numbers we published.
Why polling & specifically IVR polling is necessary
I have often spoken and written about the gap between expert opinion and public opinion and the challenges that creates for public policy decision-makers and politicians. A logical and rational electorate is ideal, but it is not always the case.
I want to be very clear – not all online panel pollsters are the same, not all IVR pollsters are the same. Panel polls conducted in numerous elections, including recently in the BC election, were excellent. The simple explanation of the difference between online poll and IVR poll results is that online skews young and progressive, and IVR skews older and more conservative. But there is more to it than that. Online panel polls are usually opt-in, have a great distribution of age, gender, income & education demographics, but tend to recruit from certain personality types that may make decisions very differently.
The “voter pyramid” as I call it shows the three different types of voters; there may, in fact, be many more as identified by Myers-Briggs or other personality research. The point is, not all voters who are the same age, gender, income, geography or education will behave in the same way, within each stratum, there will be subgroups of voter types. For simplicity, we call them head, heart and gut voters.
In my experience, non-probability sampling like online panels tends to overrepresent head voters and gives an inaccurate measure of the whole of public opinion. This certainly can be representative of informed opinion, but lower information voters still get a vote, as best highlighted with the recent US election.
When I said in 2016 that Donald Trump had won the first Presidential debate, people did not take it well. I pointed out then, and again after the election the inherent dangers of herding behaviour in the public opinion research industry and the media that report on them.
The American media that cover polling have now recommended strongly that all public opinion research be published regardless of the findings. The importance of publishing what may be considered outlier polls at the time to avoid media ridicule is now what many blame for the unpredictable Trump win. Had more polls showing Trump wins in Michigan, Ohio & Pennsylvania been published, the Clinton campaign may have redeployed resources to those States that gave Trump the 270 electoral college votes he needed to win.
Whether it is the Donald Trump victory, or Brexit, public opinion research plays a vital role in decisions made by campaigns, elected officials, industry and financial markets. The answer is not fewer polls, or censorship of various modes and methodologies. It is more polls, more methodologies and experimentation that we have long invested in.
Throughout the Calgary election, we had publicly called for the disclosure of polls that were conducted by other polling companies which should have balanced how polls may have impacted the campaign itself. This lack of polls by professional organizations with a measurable track record to compare to, might have given us reason to pause and re-examine our data. Instead, the late release of an LRT organization poll & Canadian Municipal Election Study poll gave us no reason to doubt the accuracy of our results. [1]
Suppressing publication of polls of any kind is dangerous for democracy and should be universally condemned by the industry.
Why we took it personally
A lot has been made of the tone in the last few weeks of the campaign, my own included. I have admitted that the level of debate and discourse on social media and in the media went too far. I have noted above that a number of our previous polls were attacked as inaccurate without context. It is difficult to accept criticism at the best of times, but when it is based on false information, it is especially so.
As I also noted above, we have had a long and great relationship with academics. The continued suggestion that our methodology was flawed or “unscientific” is and was unfounded. We take our work very seriously and rigorously tested samples vs. ward oversamples, and demographic over samples. The error was due to something in the frame design which might have been particular to the Calgary electorate. Dr. Angolano may decide to look at this as well.
In a past life as a campaign manager and strategist, I taught my candidates and staff to make a single comment on public opinion polls, as follows: the only poll that counts is the one on election day. The comments made on our polls was unprecedented, unfounded and unprofessional. I know it will be easy now in hindsight to say that is not true because this one poll was wrong, but the suggestion that our methodology and track record was anything but great was wrong and unfair. It fed the social media discourse, and we were sucked into the campaign in a way that was not necessary.
Most importantly perhaps beyond all of the above is the regret that I have regarding the personal relationships that were completely destroyed by the Calgary campaign. We have worked with Nenshi team members in the past, and I was especially saddened to see that some of the most vicious attacks came from people that I consider friends.
I have been in those shoes before. I understand the politics behind what they did, that is politics. No amount of money or power is worth my personal reputation. I have accurately measured and predicted Liberal, NDP, PC, Wildrose, CAQ, and hundreds of non-partisan wins without passion or prejudice; there is no partisanship or bias involved in the measurement.
Their ability to overlook all of that and the body of work I have produced the last seven years, and say what was said, is truly impressive. That is history.
What next
I look forward to reading the report from Dr. Angolano and working with him to implement any recommendations he suggests may be needed to improve our research methods.
Mainstreet Research will continue to conduct public opinion research across North America, Canada and Alberta and we are confident that we will get to the bottom of this single polling error. We look forward to the next election and remain confident our polls will continue to provide accurate snapshots of election outcomes outside Calgary.
[1] . Important to note as was pointed out by CMES that the Forum poll that was often cited as such, was not in fact a Forum poll, it was Forum recruitment for an online panel.
