Did something go wrong?

Soon after the election, on the op-ed page of The Wall Street Journal, Everett C. Ladd, who directs the Roper Center, a polling archive at the University of Connecticut, called 1996 "the pollsters’ Waterloo" because they underestimated Dole’s strength. The following April, Ladd retracted his earlier statement with regret, and said that the polls’ "final estimates in 1996 were generally good ones." In this case, as so often, the initial attack drew far more media attention than the subsequent apology.

It’s true that the polls showed inexplicable variations and fluctuations during the course of the race. Toward the end of June, for example, a Yankelovich Partners CNN/Time poll was showing a Clinton lead of 6 points at the same time that an ABC/ Washington Post poll showed a 20-point advantage.

As the campaign proceeded, different kinds of people were queried. For example, the Harris Poll began its early pre-election surveys by reporting the opinions of the general public, then switched its base to registered voters in the summer. In September and October it moved on to those who said they were "absolutely" or "very likely" to vote. Finally it went to the 50% of the original sample who were "absolutely" certain to vote and who (if eligible by age) said they voted in the last election. Needless to say, people who take the trouble to vote are not a general cross-section of the public. State and local polls use different questions to define likely voters, different sampling procedures, and different quality controls. Some are year-round operations; others are activated only for campaigns.

Many work on shoestring budgets. Polls once run by specialists on major newspapers are now, with a few exceptions, farmed out to independent research organizations.

Do polls affect the outcome?

Publication of polls stopped one week before last year’s French election because of fear that they affect the actual vote. In the United States, restricting or banning them is not an option.

As the Harris Poll’s Humphrey Taylor puts it, "Our job is to publish and be damned." Did some pro-Dole people not bother to vote because the pre-election polls showed such a big lead for Clinton? It is just as reasonable to suppose that many Clinton supporters didn’t vote because they thought his victory was a sure thing. Did the polls underestimate the Dole vote because Republicans distrust the "liberal" media, and see polls as their instrument? The more probable explanation is that Dole’s better-educated, higher-income supporters were more likely to vote than Democrats were. The social differences that explain variations in turnout are also reflected in political choices.

There can only be one winner in an election race, and political polls are judged by their accuracy in forecasting who that will be. Yet elections (as in 1960) can be won by the narrowest margin of the popular vote, and the peculiar institution of the Electoral College makes projections of the national popular vote nothing more than a preliminary indicator of the outcome state by state.

Not an exact science

Pollsters are not only expected to deliver precise estimates of the public’s choices; they must base those estimates on the choices of those who are really going to cast a ballot. This is daunting indeed in the case of primary elections to fill local offices, when only a small fraction of the electorate may turn out.

Turnout is affected by the weather and by public interest in concurring local or statewide contests. It makes a very big difference in some presidential election years but not in others. Calculations of turnout are based on the number of valid ballots cast in relation to the voting-age population, though this includes millions of aliens, recent movers and felons who cannot be counted exactly. Some states allow registration at voting time. An unknown number of enthusiasts register in more than one precinct. Between 70 and 80 percent of election poll respondents claim to be registered, but not all of them are. Many registered voters don’t vote, and there is no easy way of determining who will.

Today most survey findings are published with a brief accompanying statement about their margin of error, commonly given as plus or minus three percentage points when the response is evenly divided in a sample of 1,000. (An important qualifier is that there is one chance in 20 that the error is greater.) The margin gets smaller as the sample increases and as the response gets farther from an even division. "Error" does not mean a "mistake." It is a statistical term, based on pure mathematical probability, which assumes that the sample is perfectly representative and the survey process itself completely unflawed. In practice this never happens. Budgets set limits on sample size and on the time-consuming procedures required to make samples properly representative of their parent populations. There are innumerable lapses between plans and execution. The public has become steadily less cooperative as telephone interviewing is increasingly identified with the annoying intrusion of telemarketing.

Accidental and inexplicable occurrences can affect the top-line survey findings that find their way into the headlines. Most survey results are based on just a fraction of the people who should have been picked by pure chance. People who can’t be reached differ from those who are, in ways that may be reflected in their opinions, including their political choices. This is also true of people who are reached but who refuse to go through a whole interview. There have been more such people with each passing year.

Pollster adjustments

To compensate for sampling flaws, all the election polls — using their own special formulas — weight results to conform to the characteristics of the population, as described by the census. The trouble is that the 1990 census figures, although updated by annual surveys, share the affliction of all those who seek to extract information from an ever more recalcitrant public.

Some pollsters still use curiously personal ways of adjusting their data to conform with their hunches. This starts with the respondents who can’t answer questions, because they "don’t know," are reluctant to answer, or haven’t made up their minds. One unresolved technical question pollsters face is how to dispose of the people who claim to be registered voters and who intend to vote, but who have still not made their pick at the time they are interviewed. This can be a substantial proportion when a race is in its early stages. The very fact that polls are not unanimous presents a constant reminder of their inherent uncertainties, which journalists and the public alike seem reluctant to accept. Integrity and professional competence cannot ensure total accuracy in an enterprise that entails lots of gut judgment and thrives on dumb luck.

The polls of which one should be most wary are the ones that never get published: the research that is done by and for political consultants. The biggest operators in this domain are outside the reach of the research profession, and are not committed to the standards and codes of its associations. They do not follow the fundamental rule of explaining to all comers exactly what they do. With sometimes skimpy evidence and shabby inferences, they cultivate the notion that politicians should cater to the superficial vagaries of public opinion rather than lead it through the strength of their convictions. Editors and the public should pay less attention to the inexact art of election forecasting and more to the sinister implications of political marketing.

A longer version of this article appeared in the May/June issue of Society.

Bogart is the author of "Polls and the Awareness of Public Opinion" and "Preserving the Press." He is former president of the American and World Associations for Public Opinion Research.