Sampling Considerations Associated with Telephone Surveys and Web Panel Surveys

When initiating a research project, a key consideration is the degree to which the results are projectable to the population the sample is designed to represent. Ideally, a truly random and projectable sample is desired. To meet these criteria and be a true probability sample, the sample drawn from the population under study must allow every person in the population a chance (greater than zero) of being selected in the sample.

The reality is that the closest thing we to true probability sampling is door to door sampling. This is the approach most commonly used by the Bureau of the Census to conduct the U.S. Census. This approach is suboptimal for research for a number of reasons, not the least of which is that it is very costly.

Telephone Surveys Do Not Yield True Probability Samples

Many claim that Telephone Surveys allow for true probability samples as they are recruited from lists of people who represent a target population, and each person has an equal probability of being selected. On the contrary, Telephone Surveys exclude certain elements in the population since:

Not everyone has a listed telephone number
More people are moving to cell phones rather than landlines
Many people are included on the national “Do Not Call” list
Many households employ gatekeeper technologies (e.g. Caller ID, answering machines)

These consumer strategies not only increase the cost of doing Telephone Surveys but they also make some researchers question the extent to which the findings can be projected to the total population under study.[1]

Web Panels Produce More Reliable Data Estimates

Currently, the research industry commonly uses randomly selected telephone samples. A study conducted by the University of South Florida comparing a Web Panel Survey in which respondents were randomly selected from a panel versus a Telephone Survey that employed a “cold-calling” method to randomly select respondents found that Web Panels can produce more reliable data estimates than Telephone Surveys.[2]

Web Panels Allow for Convenience Samples

One of the key objections to Web Panel Surveys from a statistical sampling perspective is that they do not allow for a random probability design. Web Panel Surveys allow for convenience samples or non-probability samples where some people in the population under study have no chance of selection because they are not a participant in the panel. One of the main concerns around Web Panel Surveys is the exclusion of populations due to:

Age
Income
Education
Ethnicity
Population density (Rural, Suburban, Urban)

The exclusion of these sub-populations is based on the notion that these groups lack internet access. While true, this is a diminishing concern for Web Panel Surveys. In fact, research from the Pew Research Center[3] shows:

84% of American Adults have internet access
A clear majority of senior citizens, 58%, have internet access
While those with college educations are more likely to have internet access than those who do not have high school diplomas, the class related gaps have shrunk dramatically, with the most growth coming from those in lower income households and those with lower levels of educational attainment
While African-American and Hispanics are less likely to have internet access than White and Asian-Americans, the clear majority have internet access
Those in rural areas are the least likely to have internet access; however, 78% of them do

Telephone Surveys Are Susceptible to Social Desirability Bias

The Pew Research Center[4] analysis has shown that Telephone Surveys are susceptible to social desirability bias, a bias in which survey respondents answer questions in a manner that will be viewed favorably by others. The Pew Research Center found this bias to be significant in questions about family and social life, ability to pay for food and medical care, and political sentiment questions.

Selecting the Right Methodology

Against this background, when planning a research project, methodological considerations should include the benefits and detriments of both Web Surveys and Telephone Surveys. It is important to consider the population holistically, the best possible way to reach the population, and the cost implications. In some cases, it may be using just one methodology, be it Web Survey or Telephone Survey. In other cases it may be a combination of the two, whether equal weight or with one methodology “boosting” the other.

Xavier Alvarez is a Project Analyst at Q2 Insights, Inc., a research consulting firm with offices in San Diego and New Orleans. He can be reached at (760) 230-2950 ext. 4 or xavier.alvarez@q2insights.com.

[1] Struebbe JM, Kernan JB, Grogan TJ. The Refusal Problem in Telephone Surveys. J Advert Res 1986:29–37 June/July.

[2] Braunsberger K, Wybenga H, Gates A.A Comparison of Reliability Between Telephone and Web-Based Surveys, Journal of Business Research 60 (2007) 758–764

[3] Perrin, Andrew, and Maeve Duggan. “Americans’ Internet Access:2000-2015.” Pew Research Center: Internet, Science & Tech. N.p., 26 June 2015. Web. 02 Dec. 2016.

[4] Keeter, Scott. “Methods Can Matter: Where Web Surveys Produce Different Results than Phone Interviews.” Pew Research Center. N.p., 14 May 2015. Web. 02 Dec. 2016.

This entry was posted in Quantitative and tagged on December 8, 2016 by Q2 Insights