New EDF poll statistically invalid due to biased questions

Posted on November 14, 2008


Environmental Defense Fund’s latest poll claims that most Americans want climate change addressed as Congress tackles the economic problems facing the United States. But there’s a fundamental problem with EDF’s poll: the questions are so biased that the results are statistically meaningless. In addition, one poll question mentioned in the pollster’s summary slides is not listed in the EDF press release, an error that gives the impression that EDF cherry picked the results they published. Furthermore, the poll’s stated margin of error is either incorrect or indicates that the poll has a lower-than-usual confidence level. These problems raise concerns over whether the poll’s stated results are accurately described or whether they are an attempted deception on the part of EDF.

What follows below is a question-by-question analysis of why the questions are biased. The questions have been reproduced from the EDF press release, but the results have been removed for clarity.

Question #1 is fine. It simply asks who the respondent voted for. I’ve removed it since it has no bearing on the rest of the questions.

2. Given our economic situation, which comes closer to your view?

  • Investments in new, clean energy could create millions of new jobs, so now is the time to address the problems of global warming and climate change.
  • It is less important to address the problems of global warming and climate change, given the high energy prices and economic downturn in our country
  • Not sure

Question #2’s first flaw is that the question’s topic is “our economic situation”, but the two answers are about “the problems of global warming and climate change” instead. The two answers are related more to the unasked question of “when should the problems of global warming and climate change be addressed, now or later?” than to the economic situation.

The second flaw is that “our economic situation” is vague and applies to multiple components of the economy. The questions should ask about the economy as a whole or should have polled the various components of the economy – unemployment, high oil and electricity prices, the stock market and 401(k) returns, et al – independently Each of these specific components should be polled independently, yet they’ve been combined here. The structure of the answers forces the respondent to choose either “millions of new jobs” or “economic downturn”, and either “new, clean energy” or “higher energy prices,” resulting in two false either/or dichotomies.

The third flaw in Question #2 is that the language is itself biased to produce a specific response. “New, clean energy” and “create millions of new jobs” are both positive concepts, while “high energy prices” and the word “downturn” are negative concepts. This biased language psychologically pushes a respondent toward preferring the question with the most positive concepts over the one with the most negative concepts. And the response to this question was 58% in support of the positive concepts.

3. Generally speaking, do you think it is very important, somewhat important, not very important, or not at all important to address the problems of global warming and climate change?

  • Very important
  • Somewhat important
  • Not very important
  • Not at all important
  • Not sure

This question suffers from only a single flaw that I can identify, and that’s the use of slightly biased language, specifically the use of the negative concept of “problems.” People don’t like problems, and when given the choice, people will work to fix them. By defining global warming and climate change as “problems,” the question is slightly biased toward responses that will result in a fix for the “problem.” The results show this as well, with 78% saying that it was either very or somewhat important to fix the “problem.”

The larger problems is that this question comes after the heavily biased Question #2, which had already psychologically connected fixing the “problems of global warming and climate change” with positive concepts like new jobs and clean energy. The connections forged between concepts in Question #2 applies a bias to the results of all the remaining questions.

4. Which of the following would you say is the best way to create jobs and stimulate the economy?

  • Investment in building clean energy products, thus promoting energy independence and creating manufacturing jobs
  • Investing in road and bridge repair
  • Cutting government spending
  • Cutting taxes
  • None of the above
  • Not sure

The most serious flaw in Question #4 is that there is an obviously “correct” answer to the question, namely the first answer. We know that this answer is the one the pollster wants because the question asks what is “the best way to create jobs” while the first answer says by “creating manufacturing jobs.” In other words, the best way to create jobs is to… create jobs. This flaw is common in so-called push polls, and it’s this question that makes the entire poll suspect.

5. Which statement best reflects your view?

  • Congress needs to deal with the problem of America’s addiction to oil because it undermines our long-term economic future and causes pollution
  • Dealing with our energy problems will have to be delayed while Congress focuses on economic issues
  • Our oil addiction and long-term economic problems are related and Congress should deal with them together
  • Not sure

The first flaw in Question #5 is that the first and third answers are so similar that they offer a 2/3rds chance that the respondents will answer the question in a manner that is advantageous to EDF. The first answer says that we need need to fix “America’s addiction to oil because it undermines” the economy (emphasis mine). The third answer answer says that oil and economic health are related and have to be tackled at the same time. The first answer implies a relationship while the third answer states one.

The bias in these answers is focused on creating as much negativity as possible – and then offering the respondent a way out. The first answer uses the words “problem”, “addiction”, “undermines”, and it mentions “oil” instead of the more value neutral word “energy.” The third answer uses “addiction”, “problems”, and “oil.” The second answer, which stands in opposition to the first and third answers, uses the words “problems” and “energy” instead of the more negative word “oil”, resulting in the least negative option. Because the second answer is the least negative and threatening, the respondent will be more likely to choose one of the other answers that offers escape from the greater implied threat.

Finally, the first answer was likely the response that the question was designed to illicit. Not only is it the answer with the most negative concepts, and thus the answer that respondents would theoretically want to solve the most, but the first answer also tacks on the unrelated and unnecessary concept of pollution. None of the other questions discuss pollution in any way, making the words “and causes pollution” an obvious attempt to make respondents select this answer.

The first and third answers collectively got 76% of the respondents, while the second answer and “not sure” received only 23% collectively. (The sum is less than 100% due to rounding error.)

6. If Congress decides there is a need to invest more money to stimulate the economy to create jobs, what would you say is the best way to pay for it – by increasing taxes, borrowing more money by increasing government debt, or using revenue generated by making large companies pay for the global warming pollution they produce?

  • Increasing taxes
  • Borrowing more money through increasing government debt
  • Using revenue generated from large companies that pay for the pollution they create that leads to global warming
  • Not sure

The first flaw in Question #6 is the different amounts of information between the answers. The first answer is simple and direct – “increasing taxes.” It’s supposedly difficult to misunderstand, although it’s interesting that the question doesn’t say whom would have his, her, or its taxes raised, leaving the respondent to assume it means his or her taxes. The second answer is both more complicated and redundant, using both “borrowing more money” and “increasing government debt” even though borrowing more money means increasing government debt. The third answer is the most complicated because the language isn’t clear and because it brings global warming and pollution into the answer (but not “climate change,” which was last used in Question #3 above). The long and complex third answer means the questioner will spend more time on this answer than the other two (possibly combined). The result is that the respondent will remember the last answer the best and maybe biased to select it over the other two.

The third answer also uses coded instead of plain language. “Using revenue generated from large companies” means a carbon tax or a cap-and-trade system that will increase the cost of the companies’ products. Similarly, “large companies that pay for the pollution they create that leads to global warming” means utilities, chemical companies, and energy companies, all of whom are likely to pass any new carbon emissions costs on to their consumers. If the question was phrased “Pricing energy sources so utilities and oil companies pay more,” it’s easy to imagine the response rate could have been much lower than 66%.

I mentioned above that “climate change” was missing from the third answer. This is potentially a problem because it can apply a bias to the third answer. “Global warming” has been tested and verified by multiple studies to be a more pro-environment phrase than “climate change”, a fact that applies some additional bias to the third answer.

7. Okay…Which of the following is most important to be near the top of the president’s agenda in January?

  • New regulations for the financial services industry
  • Creating jobs by investing in renewable energy projects
  • Reviewing our national security strategy
  • Not sure

While this question is one of the better ones, the language is still slightly biased. “Regulations” are viewed, correctly or not, as negative concepts in the U.S. today, so the first answer has a slight negativity attached. “Creating jobs” is a positive concept for the second answer, and “reviewing” is a neutral concept. This puts a slight bias on the second answer.

In addition, the language has a disconnect between the answers – “new regulations” isn’t the same as “creating” and “reviewing”. Instead, “regulating” would be the same. The answers could have been made more similar by changing either the first answer to match the other two or to change the other two to match the first.

8. In your view, is it more important to limit the amount of regulation that is placed on business, or to work to implement new rules that make sure companies produce clean, renewable energy to promote environmental standards and energy independence?

  • Limit the amount of regulation that is placed on businesses
  • Implement new rules that make sure companies produce clean, renewable energy to promote environmental standards and energy independence
  • Not sure

The main flaw in Question #8 is that it’s a false dichotomy – it’s entirely possible to “limit the amount of regulation” on businesses while simultaneously implementing rules that “make sure companies produce clean, renewable energy to promote environmental standards and energy independence.” The two are not exclusive, especially given the vagueness of the word “limit.” Does “limit” mean no regulations, no new regulations, only a limited number or type of new regulations, or what? It’s not stated, and so the respondent’s mind has to fill in that detail.

In addition, to “implement new rules” means to regulate, and yet the second answer doesn’t actually say anything directly about “regulations” like the first answer does. This is another example of coded instead of plain language, and it further biases the results of the question.

Finally, the second answer also includes a number of positive concepts such as “clean,” “renewable,” and “independence,” concepts that bias the answer positively and make it more psychologically attractive.

9. Would you strongly favor, moderately favor, moderately oppose, or strongly oppose legislation to cut foreign oil imports and reduce environmental pollution?

  • Strongly favor
  • Moderately favor
  • Moderately oppose
  • Strongly oppose
  • Not sure

The only flaw with this question is that it combines two things that should be separated to gauge public opinion on both independently. While legislation to “cut foreign oil imports” could be tied to legislation to “reduce environmental pollution”, such legislation could be implemented independently too. Splitting this one question into two, one each for oil independence and pollution reduction, would also have gathered more information, especially if a third question was added that tied both oil independence and pollution together again.

10. Would you still support this legislation if it raised your home energy bill by $5 a month?

  • Yes, still support it
  • No, would oppose it
  • Not sure


11. What if your energy bill increased by $10?

  • Yes, still support it
  • No, would oppose it
  • Not sure

Both Questions #10 and #11 are fine so far as I can tell – no biased language, no false dichotomies, no unnecessarily complex language. But there’s a more significant problem related to Questions #10 and #11 – where is Question #12?

On page two of the summary pdf file, the bottom left slide has a statement that reads:

Only when we asked if they would support the legislation if their energy bill was raised $15 a month did the numbers flip. 42% favored [legislation that would cut oil imports and reduce pollution] and 46% did not.

Similarly, page 6 of the summary PDF file has the results of this question graphically – 42% yes, 46% no, and 12% not sure. But the questions listed in the EDF press release stop with Question #11 above – “What if your energy bill increased by $10?” The question that asked about the increase by $15 is not in the press release. Was it simply an oversight and missed in a cut/paste operation, or was it left off on purpose because the results were inconvenient? Are there other questions that were asked but not reported in the press release? This oversight creates the perception that EDF is attempting to deceive the public and/or the media regarding the poll’s results, because it’s only that question that went against the grain of the rest of the poll. And the fact that this question is missing makes even more pointed the issues of bias throughout the rest of the poll.

Finally, I’d like to address the poll’s margin of error (MOE). MOE is calculated by an equation that includes the size of the sample, the size of the population being sampled, and the desired confidence level (a measurement of how likely it is that the results are valid and not totally random). Unfortunately, while the MOE, sample size, and population size are stated in both the EDF press release and the summary slides, the confidence level of the results is not stated in either. According to an online sample size calculator, the MOE for the entire U.S. population (approximately 305 million people), a sample size of 600 respondents, and the standard 95% confidence level is 4.0%, not the 3.5% stated in the press release. In order to get a MOE of 3.5%, I had to lower the confidence level from the standard 95% to 90%, which is the lowest confidence level generally considered acceptable by public opinion researchers. This means that there’s a 1 in 10 chance that the results of a well designed, unbiased poll are totally off the wall instead of a 1 in 20 chance. Unfortunately, neither the summary slides produced by pollster David Schoen nor the EDF press release mention the confidence level of the poll results. For a further description of margin of error, see the American Association for Public Opinion Research’s (AAPOR) Margin of Sampling Error page.

Its difficult to know what the purpose of this poll was. Was it a messaging test poll that got released because the results were “just too good” to keep a lid on? Was it a poll that EDF didn’t vet well enough to root out the biases I described above? Or was it an environmental push poll designed to give cover to politicians and activist organizations working in support of green jobs programs, renewable energy and power grid infrastructure upgrades, plug-in hybrid vehicles, and carbon emissions caps? Only EDF knows the answer to those questions, and I don’t suspect they’ll be forthcoming with the answers. But this poll is so biased, so close to a push poll, that it will ultimately do EDF no good at all and could very easily be used by EDF’s ideological and anti-environmental opponents against EDF. This poll is also a hit on EDF’s credibility, something that will make EDF’s stated goal of working with businesses, government and communities to create lasting solutions to the most serious environmental problems more difficult instead of less. This poll is in the same credibility-destroying league as a poll conducted by The Wilderness Society (and analyzed by me for similar problems) back in July.

Environmental groups and activists do themselves, and the causes they support, no favors when they overreach. Every time they overreach, environmental groups like EDF make it harder for them to institute the changes they feel are necessary. Bad polls give anti-environmentalist groups an opportunity to scream about how “those leftist environmentalists just want to whip up a media frenzy and raise your utility bills and take away your SUV.” This EDF poll is just such an overreach, and while it’s not the worst poll I’ve ever seen, it’s bad enough that it will come back to bite them unless EDF does something to prevent it.

For more information on how questions should be worded and ordered to avoid bias, see the AAPOR’s Question Wording FAQ page

Posted in: Uncategorized