Earlier today I did end up getting ChatGPT Plus for a cost of USD $20 a month. That apparently will provide me better availability during high demand, faster response speeds, and priority access to new features [1]. I’m not sure if this subscription will be worth keeping. I’m about to open up the latest model GPT-4 and see what it is able to accomplish in terms of advanced reasoning, complex instructions, and more creativity. I have spent the last 6 weeks thinking about writing these polling related papers. A lot of that time and effort was spent trying to figure out what the trajectory of the current literature was and what that meant for the future of understanding and evaluating sentiment. To me polling is a sampling of sentiment on something. It’s an evaluation of respondent preference or attitude. I’m writing that explanation without using the word opinion. That word choice happened on purpose.
A lot of opportunity exists to study breakdowns in modern polling techniques. This is the start of my series of inquiries into that potential area of study. To be really clear here upfront in this analysis I believe that respondent fatigue, general unavailability, and systemic methodology breakdowns have made modern polling problematic. You can see commentary in very public news sources about people being frustrated with polling [2][3]. I’m surprised so far that more of the literature did not openly discuss the breakdown in previously solid methodologies for polling the public.
We are here working on these series of literature evaluations, because I’m principally interested in opinion polling and sentiment analysis. You could go out to the Pew Research Center to learn about polling basics [4]. They present research and polling in some pretty easy to understand ways. That visit to Pew might even send you in the direction of the American Association for Public Opinion Research (AAPOR) [5]. The outcome of that digging might point you in the direction of some methods of administering polling.
Traditional mail survey questionnaire
Phone based delivery of questionnaire
Email survey questionnaire
Web based questionnaires
I have spent some time listening to Nate Silver on podcasts over the years. One of the interesting things Nate shared was that a recent batch of polls was more accurate than expected [6]. That made me wonder if the metrology being used had improved or if they were just tuning properly based on some type of modeled expectation. I will caveat here that I read Nate’s book:
Silver, Nate. The signal and the noise: Why so many predictions fail-but some don't. Penguin, 2012.
It was interesting, but I wanted to know more about the future of polling. One of the main questions that I ended up having was about the differences between polling and modeling and what method was better and would end up being more popular over time [7].
Public attitude extraction
Indicator based tracking
Modeled behavior
Simulated response analysis
Persona based economic modeling
Sentiment interviewing
Where did I end up at the end of this analysis? Two things came into focus. First, it was pretty clear to me after digging around that a clear framework for auditing polling methodologies needs to be developed. Second, a sea change in polling is about to occur where modeling as a method or some type of supplemental grounding activity will have to augment traditional polling methodologies. We have already seen extremely convoluted weighting and error ranges become forward reaching into the public mind.
You could search Google Scholar for online polling best practices [8]. You could also take a look at direct search for “modern polling methodology” and see what shows up [9]. Only one result on the first page of that search had more than 100 citations and that was a handbook. It came in at 172 citations.
Donsbach, W., & Traugott, M. W. (Eds.). (2007). The SAGE handbook of public opinion research. Sage.
The PDF was not available over at ResearchGate or on Google Scholar. You can buy the hardcover from Sage for $215 dollars which seems a little bit out of hand. You can rent or buy the eBook from Sage as well which for a lifetime copy would set you back $117 [10].
I did prompt ChatGPT from OpenAI using the GPT-4 model to generate a paper called, "A literature review of modern polling methodology" that turned out to be interesting. My exact prompt on this one was, “write an academic paper called, ‘A literature review of modern polling methodology’”.
Oddly enough on the first attempt it produced a ton of content then had a “network error” with a more specific message of, “There was an error generating a response.” My only option to move forward was to click the “Regenerate response” box and hope for the best. It took 4 regeneration attempts to get a complete output.
Title: A Literature Review of Modern Polling Methodology
Abstract
The present study aims to provide a comprehensive review of the modern polling methodologies, analyzing their strengths and weaknesses, as well as the ongoing challenges faced by the polling industry. This literature review discusses traditional and emerging methodologies, such as random-digit dialing, online surveys, and mixed-mode approaches. Moreover, the impact of social media on public opinion polling is considered, and the potential of data mining and machine learning for future advancements in the field is explored.
Introduction
Public opinion polling plays a significant role in modern democracies, shaping political strategies, and informing policy-making processes. Over the years, several polling methodologies have been developed and refined to collect accurate and reliable data on public opinion. However, recent events, such as the 2016 United States Presidential Election and the Brexit referendum, have brought the accuracy and reliability of polling methods into question. This literature review aims to examine the current state of polling methodologies, evaluate their strengths and weaknesses, and discuss the future directions of public opinion polling.
Traditional Polling Methodologies
1.1. Random-Digit Dialing (RDD)
RDD has been the gold standard of telephone survey sampling for decades. By generating random phone numbers, researchers ensure that each household has an equal chance of being included in the survey. However, the rapid decline in landline use and the increasing preference for mobile phones has challenged RDD's efficacy (Pew Research Center, 2019).
1.2. Area Probability Sampling
Area probability sampling involves selecting a sample of respondents from a predefined geographical area. This method offers an alternative to RDD and has been proven useful for face-to-face surveys (Lynn & Jäckle, 2020). However, it can be expensive and time-consuming compared to other sampling techniques.
Online Polling Methodologies
2.1. Online Panels
Online panels consist of pre-recruited participants who have agreed to take part in surveys. These panels provide a cost-effective and faster alternative to traditional methodologies. Nevertheless, concerns about sample representativeness and potential biases remain (Baker et al., 2013).
2.2. River Sampling
River sampling is an online methodology where respondents are recruited as they visit websites, often through ads or pop-ups. This method can provide a large number of respondents quickly. However, the lack of control over the selection process raises concerns about the quality and representativeness of the sample (Mercer, 2018).
Mixed-Mode Approaches
Mixed-mode approaches combine multiple data collection methods, such as telephone and online surveys, to compensate for the shortcomings of individual methods (Vavreck & Rivers, 2008). This approach has gained popularity as a means to improve representativeness and reduce nonresponse bias. However, challenges related to mode effects and data integration persist (De Leeuw, 2021).
The Impact of Social Media on Public Opinion Polling
Social media platforms have transformed how people communicate and consume information. As such, there is growing interest in harnessing social media data to understand public opinion (Jungherr et al., 2021). Sentiment analysis and network analysis have emerged as promising techniques for analyzing social media data, but concerns about data quality, privacy, and representativeness need to be addressed (Stier et al., 2020).
The Future of Polling: Data Mining and Machine Learning
Data mining and machine learning techniques offer potential for improving the accuracy and efficiency of public opinion polling. For instance, these techniques can be employed to correct biases, enhance sample representativeness, and identify patterns in survey responses (Wang et al., 2019). However, the ethical implications of these methods, particularly with
You may have noticed that the model output stopped in the middle of a sentence. It seemed like a better idea to change up the prompt for GPT-4 and request it to outline a paper instead of writing it. That seemed like a better way to get an actual complete set of output from the model. The previous 4 network errors were actually rather frustrating. OpenAI is only allowing 30 prompts per 3 hours and having to use them up to overcome network errors was frustrating.
Title: A Literature Review of Modern Polling Methodology
Abstract:
This paper provides a comprehensive review of the modern polling methodology, focusing on the evolution of survey techniques, data collection methods, and analysis tools in the field of public opinion research. By examining the strengths and weaknesses of these methodologies, we aim to identify best practices and guide future research in the field of political and social polling.
Introduction
1.1 Background and purpose of the study
1.2 Scope and limitations of the review
Evolution of Polling Methodology
2.1 Early beginnings: Straw polls and face-to-face interviews
2.2 Telephone surveys: The rise and decline
2.3 The advent of the internet: Online surveys and panels
2.4 Mobile-based polling: SMS and mobile app surveys
2.5 Social media as a data source
Data Collection Techniques
3.1 Probability sampling methods
3.1.1 Simple random sampling
3.1.2 Systematic sampling
3.1.3 Stratified sampling
3.1.4 Cluster sampling
3.2 Non-probability sampling methods
3.2.1 Convenience sampling
3.2.2 Quota sampling
3.2.3 Snowball sampling
3.2.4 Purposive sampling
3.3 Comparing probability and non-probability sampling techniques
Questionnaire Design and Implementation
4.1 Principles of effective questionnaire design
4.2 Types of survey questions
4.2.1 Open-ended questions
4.2.2 Closed
Links and thoughts:
Top 5 Tweets of the week:
Footnotes:
[1]
https://chat.openai.com/chat
[3] https://www.nationalaffairs.com/publications/detail/the-trouble-with-polling
[4] https://www.pewresearch.org/course/public-opinion-polling-basics/
[5]
https://aapor.org/
[6] https://fivethirtyeight.com/videos/polls-havent-been-this-accurate-since-at-least-1998/
[7] https://www.majoritystrategies.com/blog/polling-or-modeling-which-do-you-need/
[9] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C10&q=modern+polling+methodology&btnG=
[10] https://us.sagepub.com/en-us/nam/the-sage-handbook-of-public-opinion-research/book228115
What’s next for The Lindahl Letter?
Week 116: A literature study of mail polling methodology
Week 117: A literature study of non-mail polling methodology
Week 118: A paper on political debt as a concept vs. technical debt
Week 119: All that bad data abounds
Week 120: That one with an obligatory AI trend’s post
If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the year ahead.
A literature review of modern polling methodology