| ...when they try to explain the meaning of any before-and-after opinion polls.
The vast majority of journalists, upon receipt of a new poll, will say something like, "Before the State of the Union speech, President X's approval rating was 50%. After the speech, a new poll shows he's up to 52%, that's an impressive/disappointing (depending on the spin they're selling) change." The better journalists have had it drummed into their heads that they must add "the margin of error for the poll is +/- 3%". But they mumble it so quickly its clear they have forgotten what it really means, if indeed they ever really knew.
A scientific, nation-wide opinion poll, such as the traditional approval rating poll conducted by Gallup, Zogby, or others, is an attempt to find out what the entire country thinks about a question, by asking only a very small (usually less than 1/100 of 1%!), representative sample of people. The reported numbers (50% and later, 52%) reflects how the sample (not the entire population) actually answered the question.
A margin of error of +/- 3% indicates that if we gathered answers to the poll question from the entire country, the true approval rating would probably (95% of the time) be between 47% and 53% for the first poll, or between 49% and 55% for the second. A 2% change is smaller than the margin of error, and is not statistically significant. The actual approval rating could have been, say, 51% for both polls. The apparent 2% change might just be a side effect of who was selected for the samples, the fact that a different investigator asked the question, different individuals' feelings about the polling process, or any number of other random factors that had nothing to do with a genuine change of opinion. In fact, the actual approval rating might actually have decreased from 53% to 49% following the speech!
The poll as constructed just isn't sensitive enough to reliably detect opinion swings of less than 6%. To get a more sensitive measure you'd have to increase the sample size. The statistical concept of margin of error is hard science and not at all controversial. I should know, I was a statistics tutor in college.
In other words, it is just plain wrong to conclude that a 2% increase in the approval rating means that people were more approving of the president after the speech!
Yet time after time, journalists from reputable outfits report such insignificant changes as front page news.
Grrr...Sigh. | 
