Calgary, December 11, 2017 – A comprehensive review of the major polling failure in the recent Calgary mayoral race that re-elected incumbent Naheed Nenshi has revealed a “perfect storm” of factors that caused the polling error, and has led to concrete recommendations to mitigate future error.
The review, led by Mainstreet Research VP of Analytics Dr. Joseph Angolano, and vetted by a panel of independent polling experts, has found absolutely no evidence that the polling errors were intentional or malicious. The recommendations have already been implemented in a new poll in the ongoing Calgary-Lougheed by-election campaign, that will be released tomorrow.
“I want to first and foremost thank Dr. Angolano and those who helped in the efforts of this review. I accept all the recommendations provided by Dr. Angolano and everyone else involved in the review process.”
“This has been a humbling experience,” said Quito Maggi, President of Mainstreet Research. “But the results of this review will most definitely make us better. Dr. Angolano’s analysis along with the advice of other experts, has been eye-opening, and a valuable lesson for Mainstreet.”
Key factors in polling error
Dr. Angolano identified five factors that together led to a near 25% deviation between Mainstreet’s last pre-election poll and the actual outcome. These included a failure to poll in non-official languages and a tendency among Nenshi voters not to respond to Mainstreet polls. But chief among the findings was the issue of young voters. The unique ratio of youth to seniors in Calgary, and assumptions about whether they would vote in substantial numbers.
Maggi added, “The ratio of young people in Calgary, and a few other western urban centres is unlike anything we see in other parts of the country. We missed a lot of these young voters because they are harder to reach. Even those we did reach had significant response bias as revealed by the report. Like the election in Calgary in 2010, the increase in youth turnout produced a result not remotely close to pre-election polling. It is so unusual that Dr. Angolano has dubbed it the “Calgary Effect” due to what appears to be a repeat of the 2010 results and polling error.”
Communications review
Mainstreet has already implemented recommendations in the report to address the issues that led to the error. A review of the communications around the election is equally critical of the social media and of the tone of debate and Quito Maggi specifically.
“In the heat of an election race, myself and some of my staff were not as respectful as we should have been,” added Maggi. “We received more than a healthy dose of skepticism and critique, we also faced allegations — on social media and elsewhere — that our intent was malicious, or that our polls were rigged. We took it personally and responded emotionally. We will be changing the way we communicate so that we let our research do the talking in future. I’ve spoken to the people involved and apologized for my behavior and I take full responsibility for the behavior of my staff as well and apologize on their behalf.”
“I am accepting all the recommendations from Justin Ling and thank him for his work. Rather than trying to parse words and argue about who-said-what and why, I intend to do as he recommends and just accept total responsibility.” He said. “Dealing with the increasingly acrimonious world of politics is tough for anyone, and we’ve also faced our fair share of attacks and trolling on the internet. That has certainly led to many sleepless nights and restless days since the election. Given the personal and professional impact, it’s our hope going forward that we can improve the tone, instead of dragging it down.”
The future of polling
This review has cast a harsh light on some of the shortcomings of IVR (Interactive Voice Response) in polling. Although Mainstreet still believes it is viable with some adjustments detailed in Dr. Angolano’s report, declining response rates and a growing youth demographic that are harder to reach, mean that IVR is not the future of polling. In addition to implementing the short-term recommendations of the report, Mainstreet is actively working on next-generation polling technology and continuing to strive to improve in every way.
Mainstreet Research is actively pursuing partnerships with Academics, other researchers, and industry to implement and test new technology in the new year. This effort will require significant investment in Canada and the US over the coming 18 months. Details will be released in early 2018.
—
Message from the President
I have had the tremendous pleasure for the past few days of being astonished numerous times as I review the findings of Dr. Angolano’s technical review of the Calgary mayoral polling. I’ve also had flashbacks to both the campaign period and post election fallout in the days preceding election day by reading the Calgary mayoral polling communications report by Justin Ling. First and foremost, a big thank you to both Dr. Angolano and Mr. Ling for their great work.
As promised, I am accepting all of their recommendations, and those that have not already been implemented, will be in the days ahead.
I also want to thank everyone who helped Dr. Angolano by providing input to the review process and providing feedback on the report and data. This includes many people but primarily Professor Jeffrey Rosenthal, Professor Philippe Fournier, Professor Claire Durand, Dr. Kimble Ainslie and Professor Bryan Breguet. Thank you for your time and effort, and for helping us get to the bottom of this error. Your findings will help make Mainstreet Research better and I am very grateful for your assistance.
I want to thank the rest of my staff for their patience the past few months as we unpacked what went wrong in Calgary, and I took a delayed paternity leave to be with my newborn son. I know it has been difficult to manage the workload while I was away, and I am grateful for all the extra time everyone has put in.
Some people have suggested that we should have known these results were incorrect due to the unusual breakouts compared to conventional wisdom. They might be right. Dr. Angolano’s report is not conclusive in that regard, but I must acknowledge the possibility. Smart IVR usually simulates a likely voter model, when turnout remains within reasonable expectations, but this time it was widely off the mark. In the recent BC election, similar results in the 18-34 demographic (showing a BC Liberal lead over the NDP) was similarly unusual for the conventional wisdom, and we acknowledged it, but the overall result was still correct. Although the review proved inconclusive on this aspect, the stratified dial likely was the culprit for the unusual breakouts, as revealed by the attempted youth over sample of October 14th. As a result, stratified dials will be suspended until further testing can be done.
When the results came in and it became very clear we were wrong, I apologized and promised we would get to the bottom of this error. We have done exactly that with the help of many people. The findings of Dr. Angolano’s review are conclusive and we now know we can mitigate for each factor that led to the error. It doesn’t mean we will never be wrong again, but the changes recommended by Dr. Angolano and his panel of advisors will make it far less likely and reduce the deviation of future polls. I want to reconfirm my apology once again in general to the people of Calgary, and to a few people more specifically below.
First, to the people at the Calgary Herald, Calgary Sun, and the entire Calgary media. Sorry for putting you through another polling error like the one from 2010 which we did not participate in. You deserve better and I regret any embarrassment it caused the media and Postmedia specifically. We’ve learned a very valuable lesson with this review. The perfect storm that was the Calgary frame continues to astound me every day and I will carry its lessons with me. It will make me much more vigilant, cautious and undoubtedly both a better pollster and a better person.
To those who were were critical of our polls, especially Duane Bratt, Brian Singh, Melanee Thomas and others, I deeply regret anything I said that may have offended you, or caused you distress of any kind. You were right, public shaming and ridicule has no place in the public opinion research discourse. I promise to take that lesson to heart.
To the mayoral candidates, all of them but primarily Andre Chabot, Bill Smith and Naheed Nenshi. Our polls did have an impact in your election and I apologize for the unintended effects caused by the widely divergent polls.
Andre Chabot was perhaps most significantly impacted as the polls (ours and the other two) likely caused a polarization of support to either Naheed Nenshi or Bill Smith. I don’t believe that polling directly affects public opinion like some, but no doubt the scenario in Calgary did create strategic voting that impacted your result. For my part in that, I apologize.
Bill Smith, our consistent polling showing you with a lead, may have made your supporters complacent and affected your turnout negatively. By setting the expectation of your victory, the true significance of your achievement was lost, the third largest vote count in Calgary mayoral election history, and getting 44% of the vote against Naheed Nenshi was unimaginable just a few short weeks before the election. I congratulate you and your team on the accomplishment and deeply regret how our polling may have affected your result.
It is clear from the results and the data from the review that while our math was correct, and methodology was sound, there were multiple issues with the frame design that simply did not let us correctly capture the true support of Naheed Nenshi. Our attempts to make the frame design more representative, had the reverse effect. It’s clear from the trend lines that you had a tremendous comeback in the closing weeks of the campaign, and the significance of that achievement may have been lost in the drama of the polling error. My sincerest congratulations to you and your team again on a great comeback win and I sincerely apologize for that accomplishment being overshadowed by the polling error.
Lastly, the publication of poll results for the Calgary-Lougheed by-election tomorrow, is intended to test our ability to mitigate the challenges of the Calgary frame. Our disclosure and transparency in this process far exceeds national and international standards, and will begin a new chapter at Mainstreet Research that we hope will help regain the trust of media and the public. While we will make no claims on its accuracy beyond what is in the report, the testing of Dr. Angolano’s recommendations is an important final step in this process and I am confident this test will validate his findings and recommendations.
Report on the Calgary Election by Dr. Joseph Angolano (link)
Report on the Calgary Election by Justin Ling (link)
Calgary Q&A
It has been an exciting few days here at Mainstreet as we unpack the results of out review of both technical and communications issues related to the Calgary election polling.
There have been a lot of good questions raised, about the technical report that WE will attempt to answer below in the form of a Q&A. None of these are directed at any single individual, just a summary of good and valid questions raised so far on social media.
Q1– Is your internal review the end of the story?
No. Dr. Angolano had complete discretion to write his internal report and it was reviewed by outside experts. They included.
- Professor Claire Durand, University of Montreal & President of World Association of Public Opinion Research (WAPOR)
- Professor Jeffrey Rosenthal, University of Toronto & Fields Institute Fellow
- Professor Philippe Fournier, University of Montreal & QC125 Poll Analyst
- Professor Bryan Breguet, Langara College & Too Close to Call Poll Analyst
- Kimble Ainslie, President Nordex Research
The MRIA announced they will be launching their own review in the new year. We are excited to hear from the MRIA and look forward to working with them. Our results are quite conclusive, but that is only half the mystery. How did the other mode polls do, what factors led to their errors? Now that we know why we went wrong, we are eager to see the other side of the coin. We suspect some of the same response and non response and social desirability factors had a hand.
Q2: Is it really a perfect storm when only Mainstreet made a mistake?
Yes, it is for IVR. For others, we don’t know yet. The better question might be, “why were other polls so over representing Nenshi support, and under representing Smith support.”?
Q3: Why is it that you do not clearly state the cell phone to landline ration in your frame? And shouldn’t you be weighing for this ratio?
Truth is, we don’t know for sure. While Smart IVR does detect the type of connection that is made, it is not perfect. The average ratio of our landline to cellphone is 70-30 and we know that the Calgary Ward model was closer to 80-20. Number transportability has caused some challenges over the past decade. Cell phone numbers can be ported or forwarded to landlines, landline numbers can be ported or forwarded to cellphones.
There is no definitive landline only, both landline and cellphone, and cellphone only demographic data available for Calgary that we are aware of. Admittedly, this could have a huge impact on weighting, it is impossible to know for certain. The AAPOR ad hoc report on the 2016 US election states that cell phone/landline ratio had no effect on accuracy. Our hunch is that the effect is mild, but this is something that we will be looking at in the coming months.
Q4: Why is that the other polls caught Nenshi’s support and you didn’t?
Dr. Angolano’s report on focused on the second half of this question. This is certainly the most interesting question raised. The answer is, we don’t know for sure, but there are a number of possible factors that WE will raise below.
- Online panels were able to accurately predict the 2013 Mayoral Calgary election in 2013 as referenced in Dr. Angolano’s report.
- Phone polls produced in the 2010 Mayoral Calgary Election were widely off the mark, in fact by more than in 2017, most especially in capturing the Nenshi support
- It is possible that phone polls, whether live as in 2010, or IVR as in 2017 is inherently unable to reach Nenshi supporters for reasons
- The two online panel polls in 2017 under represented Smith by 8%, and over represented Nenshi by 5-6% on average, both polls showed Chabot at approximately 4%. Some have suggested the Chabot decline as pointing to the reason for Smith over performance on final results but that math simply does not add up. Chabot finished with 3% of the vote, a 1% decline for Chabot, cannot be the reason for an 8% increase in Smith support, it is mathematically impossible.
- It is important to point out that the pollster for the Bill Smith campaign, Pantheon Research, did produce polls for a third party in Calgary that showed similar results. It is also important to note that references during the campaign to “IVR calibration” by other pollsters in the Calgary election suggested similar numbers to those Mainstreet published. Those polls were never published for reasons unknown to Mainstreet or myself.
- The MRIA review will hopefully be able to get to the bottom of this, and get access to all polls that were produced, and not only those which were published.
Q5: Why did not you not use Random Digit Dial (RDD) to begin with?
In a smaller frame like Calgary, it would not be part of our normal frame design for a number of reasons. First and foremost, because RDD reaches many business phone lines that would have undesirable consequences on the results. This can be mitigated by sampling in the evenings and on weekends as will be our practice moving forward. Ultimately, the decision to change to a Ward model eliminated the possibility of an RDD component, because we cannot pinpoint RDD to within a Ward precisely, and asking the question or screening with a “Which ward do you live in” question may lead to an over representation of high information voters. (This is our hunch on why the CMES poll over represented Nenshi and under represented Smith).
Q6. Was it really a comeback victory for Nenshi?
This reference is to my statement and based on a few factors that are in Dr. Angolano’s report and supported by other factors.
As the internal report points out, the results show a 4% lead for Nenshi over Smith on the Advance Poll and an 8.6% lead on election day results. Organizational capacity can make a difference in turnout, the volunteer and experience advantage of team Nenshi was substantial. If as speculated by some, Nenshi was leading throughout, that capacity would have likely translated to a much larger gap in the advance polls. We actually have a few very recent examples to illustrate this pretty clearly. In the 2017 federal by-elections for which we have poll by poll results, in each case, the incumbent party either maintains or increases their vote share in Advance poll versus election day results. When we look at the Calgary results from 2013, Naheed Nenshi out performs his election day results by about 2% in the Advance poll. It is important to note that in the 2010 Calgary Mayoral election when there was no incumbent, Barb Higgins won the Advance Vote and came in 3rd on election day, Rick McIver was a close second in Advance and dropped by 2% share on election day for a second place finish, and Naheed Nenshi was 3rd in Advance and out-performed on election day by 12% to win.
Although our measure was certainly off the mark on the base, the trend lines of the three “like” polls reveal a nearly 13% decrease in the gap between Smith & Nenshi over the last two weeks. Even if you only consider the two published polls, it is still a 6% decrease in that gap but of course we can’t just look at the two polls because we have the math of the third in front of us, that data was also released yesterday.
It is also important to point out that this statement was made as an apology to Mayor Nenshi and congratulates him on his re-election.
Q7: Other pollsters have had some success blending online and IVR polls, why did Mainstreet not do that?
Mainstreet has been experimenting with online/IVR and blending since 2015. I in fact have acknowledged the great pioneering work done by Frank Graves at Ekos Research on this. However, Mainstreet had evidence from their Fields Institute study that suggested blending (based on real data collected in the 2015 Alberta election, 2015 Federal election, and 2016 Manitoba & Saskatchewan elections) would create more, not less error. A major concern with online panels that we still have is its lack of probability sampling, so much so that MRIA and AAPOR insist that polls that use online panels do not report a margin of error (because their sample is not constructed at random). This is an evolving technology and we acknowledge that other pollsters and researchers may have more success with blending.
Q8: Mainstreet said that the BC Liberals would win a majority in British Columbia? Wasn’t that a warning that you can’t predict election results?
Mainstreet predicted another BC Liberal majority in 2017 based on the numbers it had for popular vote and past vote efficiencies. Just 9 votes in a single riding separated Mainstreet from that prediction being accurate. The popular vote numbers produced by Mainstreet and all other pollsters in the field were within the margin of error in the BC general election.
That said, the BC election may have contributed somewhat to the over confidence of Mainstreet on the Calgary polling. The 18-34 results in BC mirrored the Calgary breakouts that were criticized, rightly, but those toplines were correct in the end despite unusual results in the 18-34 breakout.
Q9: One of your recommendations is to add Random Digit Dial (RDD), does this mean that previous Mainstreet polls do not use random samples?
The recommendation to add RDD to all future polls is to ensure that the numbers we are connecting with and the ones we are not connecting with are behaving similarly. We have, and many others have, conducted extensive testing on this and have always found that adding RDD to call other numbers that do not exist in our frame, either makes no difference, or increases the possibility for error. It is important however, as Dr. Angolano’s report points out, to test this in each and every election poll.
However, it does not mean that our polling is not random. It is random selection within the directory frame of Calgary, some 488,000 households, and over 650,000 eligible voters, and the order in which the phone numbers are dialled is randomly determined. Random dialing within a directory is still random selection.