When initiating a research project, a key consideration is the degree to which the results are projectable to the population the sample is designed to represent. Ideally, a truly random and projectable sample is desired. To meet these criteria and be a true probability sample, the sample drawn from the population under study must allow every person in the population a chance (greater than zero) of being selected in the sample.
The reality is that the closest thing we to true probability sampling is door to door sampling. This is the approach most commonly used by the Bureau of the Census to conduct the U.S. Census. This approach is suboptimal for research for a number of reasons, not the least of which is that it is very costly.
Telephone Surveys Do Not Yield True Probability Samples
Many claim that Telephone Surveys allow for true probability samples as they are recruited from lists of people who represent a target population, and each person has an equal probability of being selected. On the contrary, Telephone Surveys exclude certain elements in the population since:
These consumer strategies not only increase the cost of doing Telephone Surveys but they also make some researchers question the extent to which the findings can be projected to the total population under study.
Web Panels Produce More Reliable Data Estimates
Currently, the research industry commonly uses randomly selected telephone samples. A study conducted by the University of South Florida comparing a Web Panel Survey in which respondents were randomly selected from a panel versus a Telephone Survey that employed a “cold-calling” method to randomly select respondents found that Web Panels can produce more reliable data estimates than Telephone Surveys.
Web Panels Allow for Convenience Samples
One of the key objections to Web Panel Surveys from a statistical sampling perspective is that they do not allow for a random probability design. Web Panel Surveys allow for convenience samples or non-probability samples where some people in the population under study have no chance of selection because they are not a participant in the panel. One of the main concerns around Web Panel Surveys is the exclusion of populations due to:
The exclusion of these sub-populations is based on the notion that these groups lack internet access. While true, this is a diminishing concern for Web Panel Surveys. In fact, research from the Pew Research Center shows:
Telephone Surveys Are Susceptible to Social Desirability Bias
The Pew Research Center analysis has shown that Telephone Surveys are susceptible to social desirability bias, a bias in which survey respondents answer questions in a manner that will be viewed favorably by others. The Pew Research Center found this bias to be significant in questions about family and social life, ability to pay for food and medical care, and political sentiment questions.
Selecting the Right Methodology
Against this background, when planning a research project, methodological considerations should include the benefits and detriments of both Web Surveys and Telephone Surveys. It is important to consider the population holistically, the best possible way to reach the population, and the cost implications. In some cases, it may be using just one methodology, be it Web Survey or Telephone Survey. In other cases it may be a combination of the two, whether equal weight or with one methodology “boosting” the other.
Xavier Alvarez is a Project Analyst at Q2 Insights, Inc., a research consulting firm with offices in San Diego and New Orleans. He can be reached at (760) 230-2950 ext. 4 or firstname.lastname@example.org.
This entry was posted in Quantitative and tagged on December 8, 2016 by Q2 Insights