Volver a Recursos

Habla como un profesional: Cómo interpretar una encuesta y comprender la investigación de opinión.

El paciente avanza

Los conceptos básicos sobre encuestas e investigación de opinión para que puedas interpretar una encuesta como un profesional.

Why this matters

Public opinion research is one of the best tools we have to understand what people think, how widespread those views are, among whom, and why. But it is imperfect, and many factors influence the validity of data. Understanding how polling works along with its limitations and potential sources of error helps ensure it is used and interpreted responsibly.

5 Things to Know About Any Public Opinion Research

  1. Polling gives us evidence that we can use, but it shouldn’t lead us astray. Too often, public opinion data is relied on to determine what is achievable and worth pursuing. Data may help us understand how members of the public feel, what they know, or how they may vote. It can reveal a gap in support that needs to be addressed, but it shouldn’t dominate policy-focused decisionmaking. Advocates and policymakers should stand in their values and shape public opinion rather than simply follow it.
  2. Establishing the purpose helps build a sound survey. This requires a clear outcome in mind. Will the poll be shared publicly? Kept private? Know whether the poll is a fact-finding mission or bolstering an established goal.
  3. Who is paying? It’s important to know whether the entity paying for the poll shares the same policy goals as those in the coalition. It will fall to advocates to sell the policy later as well as manage its effects on their communities. This can also impact how the poll is designed, who has decision-making power, and how the poll is interpreted. That’s why understanding the money behind the research is important in every polling project.
  4. Sources of errors are red flags to consider carefully. There are many types of errors in polling that impact results. These are detailed in a section below on sources of errors. In short, polling is not an exact science and if errors are not carefully considered in the analysis of data, it can mislead people into believing one set of mistruths.
  5. Advocates serve as a check and balance in the polling process. While pollsters are technical experts, advocates are issue experts. This relationship should be collaborative and it is normal to challenge each party’s perspectives in order to gain new information that might influence advocacy messaging and approaches.

There are three kinds of lies: lies, damned lies, and statistics.

(Commonly misattributed to Mark Twain)

Assessing Polling Context

Context is important to any polling data. Is opinion research being done to influence or to measure public opinion? Who is paying? What is their agenda? Does it align with your values and goals? Here are some questions to consider.

  • Purpose of the polling: are there foregone conclusions or is this a blank slate?
    • Persuasive advocacy: Primarily used to persuade policymakers that people support a certain policy. Also used by advocates to change minds and make a case within an advocacy coalition.
      • For example, Advocate X believes an abortion policy with a viability limit is the only path to passing abortion legislation. Therefore, Advocate X builds questions in a survey that affirms public support viability limits.
    • Shaping narrative: providing proof points or tested messages to support your narrative (building a poll after your desired outcome is already established).
      • For example, Organization X wants more people to eat spinach (desired outcome), so Organization X builds questions that investigate ideas and messages that lead to more positive answers about spinach.
    • Fact finding: investigating people’s attitudes and perspectives; or to explore what people want and/or need (building a poll that has no previously established desired outcomes).
      • For example, Organization X is investigating how people feel about the color green, so Organization X builds questions that investigate people’s perspectives on the color green.
    • Who is paying?
      • What is the funder’s purpose? (see above: purpose of the polling)
      • Does the funder’s purpose align with yours?
    • Public or internal use?
      • Public release: will influence what you ask and whether you’ll want those answers shared publicly.
      • Internal release: might give you more freedom to explore the facts without the pressure of a public release. Internal polling can inform campaign language and tactics without giving your opposition information.
    • Kinds of pollsters
      • Political pollsters (e.g. Impact, GSG, Breakthrough)
      • Researchers (e.g. Pew, PerryUndem, Marist)
      • Media pollsters (e.g. Gallup, NYT, Washington Post)

Terminology

  • Construct Validity: a method of testing validity; sometimes using follow up questions to either affirm or expose
    For example: I want to measure whether person X likes hot or cold beverages. Person X tells me “I like cold beverages”. In a follow-up, I ask whether Person X prefers a cold or hot beverage on a wintry day? Person X answers “I like hot beverages.” If I stopped at the first question, I’d be able to tell the world that person X likes cold beverages, and therefore should not be served hot beverages. But the validity is weak because it turns out, it actually depends on the context of the question asked.
  • Crosstabs: answers broken down by demographics. You can ask the pollster to see the cross-tabs to further understand how demographic groups (race, gender, political identity, geographic location etc.) answered each question. (Polls are usually presented as “toplines”, which are totals, while crosstabs break down answers by demographics.)
    This can help you understand how people from a certain state answered the survey or how people of a certain racial identity answered the questions. It can also help you understand the political identity of respondents and whether they are likely to vote with your topic in mind at all.
    Keep in mind that the subgroups in cross-tabs may be too small (where the N value is small) to tell you anything reliable. As the sample group gets smaller, the potential for error increases.
  • Margin of error (MoE): predicts the likelihood of errors in the representation of a group in the sample size. (see sources of errors). While MoE has become a proxy for total survey error/precision, it is actually a measure of how accurate the sample size might be and relies on the premise that everyone had an equal chance of participating (see Sources of error > Recruitment methods; MoE does not account for other sources of errors).
    MoE grows in oversampled groups. Smaller groups leave higher probability of sampling error. Use this calculator to calculate sampling errors.
  • N value: refers to the total number of respondents or participants.
  • Qualitative Research: a research approach that involves open-ended questions, where participants can answer in their own words. Qualitative research involves listening, observation, and interpretation. Most often in the form of focus groups, in-depth interviews, journaling, and ethnography (think vibes, not hard numbers). These are often smaller groups of people.
    For example, how do abortion bans make you feel? vs. do you like abortion bans? One encourages a descriptive answer (qualitative), the other asks for a yes or no answer (quantitative).
    Note: qualitative messaging is critical for messaging and can be more informative than quantitative surveys.
  • Quantitative Research: a research approach that involves closed-ended questions where participants are restricted to one of a limited set of possible answers. Involves measuring and testing. Most often in the form of surveys or “polling” (think hard numbers and percentages, less vibes). These are much larger pools of people than qualitative research.
    For example, yes or no questions or giving pre-canned answers from which respondents can choose.
  • Validity: measures how well a question captures accurate perceptions/opinions (i.e. are we measuring what we think we’re measuring?)
    Questions to ask a polling team: Are there additional data points that affirm this perception/opinion? Are there additional data points that contradict this perception/opinion?

Sources of error

Polling is imperfect and each of us plays a part in preventing unforced errors. It’s important to acknowledge that some of these errors are impossible to prevent and are listed here simply as a reminder that polling should not be applied definitively. Here are some things to watch for:

  • Sampling bias for survey participants. Impacts how representative the sample is of the population. Remember there are those who are systematically left out of the sample because of non-response bias and other errors.
  • Non-response bias. Who responds to a poll; who doesn’t respond. In the 1980’s and 1990’s, the majority of people used a landline, giving pollsters a reliable and equal way to reach survey respondents. In the age of cellphones, text messages, and emails, pollsters report a drop in responses from potential polling participants from 50% (in the 80’s and 90’s) to 1% today. Polling often skews in the direction of a small group that pollsters can reach.
  • Wording questions. (unconscious or conscious bias, stigma, conventional wisdom). Does your question lead? Does it put a thumb on the scale? Does it play into prevailing narratives familiar with the general public?
  • Reading level. This is critical and generally under-considered. Readability ensures questions are accessible and understandable to people across varying degrees of competency. Without attention paid to readability, we risk undermining the credibility of our results
  • Order of questions. Does one question prime them in a certain direction on later questions?
  • “Mode effects”. How the survey is administered (phone v. online) can influence how a person answers the questions
    • For example, it might be harder for a person to be honest with another human on the phone vs. a digital interaction.
  • Social desirability bias. A bias that refers to when a person is influenced by how they want to be perceived. This is similar to how peer pressure might change what you say and do for the sake of wanting to be liked.
    • For example, Taylor Swift is an internationally popular superstar but you really don’t think her music is that great. When a friend asks you at a dinner in front of a group of Swifties if you also love Taylor Swift, you say “I can definitely see why she’s iconic”. My desire to be perceived as a Swift enthusiast — instead of my honest opinion — impacted my answer.

 

Investigación de opinión sobre el aborto tardío y las prohibiciones del aborto.