Research methodology under the microscope: the findings from the UK polls inquiry

By Peter Mouncey, Editor in Chief, International Journal of Market Research (IJMR) Following on from the inaccurate predictions of the published polls throughout the UK 2015 general election campaign, the British Polling Council (BPC) and the Market Research Society (MRS) launched an inquiry into the performance of the pollsters last July. I reported their initial open meeting in my IJMR Editor’s blog at the time (https://www.mrs.org.uk/ijmr_blog_archive/blog/1298). The details of the inquiry itself can be found on the BPC website (http://www.britishpollingcouncil.org/tag/inquiry/). The interim findings were presented by the inquiry team on January 19th 2016, and the final report was published on March 31st. (http://eprints.ncrm.ac.uk/3789/).

The key factor: unrepresentative samples

Unrepresentative samples were the key factor.  In particular, having over 65s as a single group in an aging population is no longer sophisticated enough. Online and telephone modes under represented those older people who are likely to vote, and older electors have a higher propensity to vote, which led to undercounting the Conservative vote.  Young electors are more likely to say they will vote, but less likely to do so – over-representing likely Labour support. Secondly, those who are more politically engaged are more likely to participate in polls. Pollsters face low response, self-selecting samples – one pollster stated that 30,000 calls were necessary to achieve 2000 participants. Hard–to-contact groups really matter. The inquiry team split the face-to-face random probability samples from two post-election surveys to examine voting behaviour:  those interviewed after one or two attempts closely mirrored the pollsters’ findings; by adding in those contacted in subsequent attempts the result was closer to the actual election result. Other factors Other possible causes had no significant impact, if any at all. ‘Herding’ was also examined as the final polls were tightly clustered. This was due to weighting models being fine-tuned throughout the campaign, but there was no evidence of collusion. Applying the initial weighting schemes to the final polls reduced clustering but didn’t improve accuracy. ‘Flawed, but better than nothing’ (Guardian editorial 20th January) Recommendations cover: actions for BPC members to address the key flaws identified by the inquiry, with an audit before the next election to identify the actions being taken; changes in BPC rules to create increased methodological transparency;  a face-to-face random probability poll during the 2020 campaign; communicating the inherent imprecision within polling.  Some pollsters have undertaken their own investigations, and were transparent in meeting the inquiry teams’ requests for information and data. Limitations on polling activity are unlikely.  Stakeholders deserve to be informed.  Restrictions on publication will lead to the public being uniformed whereas others will still fund private polls. Clients must take some responsibility if ‘fast & cheap’ equals ’quick & dirty’. Methodology is under the microscope, with the potential negative impact on trust in research.  Going forward, ‘fit-for-purpose’ must surely be a key consideration, but as the inquiry warn: ‘….there will be no silver bullet for the problem of polling errors’.  Peter Mouncey FMRS F IDM A Visiting Fellow at Cranfield School of Management, specialising in key account management, market research and marketing accountability. Appointed Editor-in-Chief of the Market Research Society’s (MRS) ‘International Journal of Market Research’ (IJMR) in 2004. Peter is a joint editor of ‘Market Research Best Practice, 30 visions for the future’ (ESOMAR/Wiley 2007) and joint author of ‘Marketing Accountability’ (Kogan Page 2009) and ‘Marketing Value Metrics’ (Kogan Page 2014). He was also a Finalist for the MRS Silver Medal in 2004. Peter is a long term member of the MRS Market Research Standards Board and runs seminars on data privacy and research for the MRS, SRA and in-company. From 2000 until retirement in 2013, he was also Director of Research at the Institute of Direct Marketing (IDM), developing courses covering CRM and Data Management, teaching market research on their Diploma course, and a member of the Executive Board of the IDM journal (Journal of Direct, Digital and Data Marketing Practice). He was also founder member of the IDM Data Council. Worked at the Automobile Association from 1971-2000, latterly as General Manager, Group Marketing Services & CRM Strategy. Fellow of both the MRS and IDM. By Peter Mouncey, Editor in Chief, International Journal of Market Research (IJMR) Following on from the inaccurate predictions of the published polls throughout the UK 2015 general election campaign, the British Polling Council (BPC) and the Market Research Society (MRS) launched an inquiry into the performance of the pollsters last July. I reported their initial open meeting in my IJMR Editor’s blog at the time (https://www.mrs.org.uk/ijmr_blog_archive/blog/1298). The details of the inquiry itself can be found on the BPC website (http://www.britishpollingcouncil.org/tag/inquiry/). The interim findings were presented by the inquiry team on January 19th 2016, and the final report was published on March 31st. (http://eprints.ncrm.ac.uk/3789/).