veryday, or so it seems, we read or hear reports of the results of public opinion polls or surveys.¹ Whether it be a national poll providing the latest approval ratings of the president, a statewide poll reporting the results of the latest trial heat in the race for governor, or a local survey describing the public's evaluation of local government services, results from surveys have increasingly become a part of the political process in this country. This article explores the rationale underlying the role of public opinion, describes some of the ways in which public opinion data are used, outlines a number of the key elements of a quality survey, and cites some of the characteristics that a good consumer of public opinion data should look for in evaluating the results of a survey.

THE IMPORTANCE OF PUBLIC OPINION            

In a democratic system of government, the ultimate political authority rests in the hands of the people. Yet given the number of citizens in the country, the state, or even the smallest municipality, democracy as practiced in the ancient Greek city-states – in which the legislature was comprised of all citizens, who voted directly on all issues – is impractical.   

In our representative democracy, therefore, it is the responsibility of government officials to carry out the desires of their constituents. The challenge for public officials, of course, is determining “what the public wants.” While an elected official can sometimes vote in a way with which a majority of his or her constituents disagrees, those who desire to remain in office for any length of time generally must consider the views of those who elected them. Office-holders who consistently vote against the wishes of the people of their districts rather quickly find themselves to be former elected officials. But how do representatives who want to reflect the desires of their constituents discover what these preferences are?

There are a number of ways that public officials attempt to determine the views of those they represent. One is through the election of office-holders. While a number of factors enter into the vote decision, candidates who are successful in their bids for office can have some confidence that their constituents support their positions. The use of initiatives and referenda provide a more direct means of determining the public’s views on specific issues in that these procedures involve votes by the people on laws or constitutional amendments. Public rallies and demonstrations also provide an indicator of issues in which at least an interested minority of the population is concerned, and attendance and support of such demonstrations can aid officials in determining the public’s views. Opinions expressed at public meetings can also aid office-holders in assessing the views of their constituents, as can personal meetings, letters, telephone calls, or e-mails to the representative, or letters to the editor. Elected officials also use contact with interest groups and discussions with community leaders or other office-holders in an effort to identify the public’s views.

While each of these methods can provide some measure of citizen sentiment, or at least how some portion of the electorate feels about an issue, they all have some limitations in terms of providing an adequate picture of public opinion. Depending on direct contacts does not necessarily provide an accurate account of the public’s wishes because such contacts are likely to be from individuals who have a more intense interest in a topic and may not represent the general public’s view. Similarly, letters to the editor are typically from people who have enough interest in an issue to take the time to write and, like other media reports, reflect the editor’s view of “what’s important” and “what the public thinks.” The views of other officials are shaped by their own filters of the electorate’s desires, which may not necessarily reflect the majority’s wishes.

The results of a public opinion poll overcome these limitations in that they provide an accurate reflection of the public’s views. By collecting information from a representative sample of some population – whether it be the country, the state, a local government, or some other group – public officials can be confident they are making decisions based on the wishes of the entire population.

HOW PUBLIC OPINON DATA ARE USED

In describing how public opinion data are used, perhaps the obvious starting point is their use in election campaigns. Polling data are used throughout the campaign cycle and the polling that is done in a campaign varies considerably according to the resources available.  Polling done for a presidential or gubernatorial campaign will generally be more extensive than that done for local candidates because of the greater resources available.

Many campaigns will conduct a poll early in the electoral process that can be used as a benchmark for tracking the campaign's progress. Such a poll is designed not only to determine the candidate's name recognition and how they fare in “trial heats” against potential opponents, but also to identify issues and problems that the electorate views as important and to determine if there are significant distinctions among subgroups on these issues. These results aid candidates in developing their campaign strategies and deciding which issues should be stressed, what candidate characteristics should be emphasized (as well as those that should be downplayed), and where opponents appear most vulnerable. Depending on resources, additional surveys are conducted throughout the campaign in an effort to monitor a candidate's standing and to determine whether any changes in campaign strategy are needed.            

The use of polling data by the media during a campaign overlaps to some extent with that of candidates, although the media are focused more on the “horse-race” elements of the campaign. Given the public’s interest in “who’s winning,” the media spends considerable resources during election campaigns to conduct trial heats and report the likely winner “if the election were held today.” Between elections, the media also provide periodic measures of presidential or gubernatorial approval as well as conducting surveys on policy issues that are perceived to be important. Such polls are frequently used to “set the agenda” and attempt to convince public officials to focus attention on a particular issue. These results are also considered by policymakers in deciding what steps, if any, to take in addressing an issue.           

While reports of media polls are sometimes the source of information for office-holders in considering the electorate’s perspective on policy issues, public officials frequently use more custom-designed surveys to assess the public’s views. Probably the largest “consumer” of polling data is the president. Franklin D. Roosevelt was the first president to have a regular flow of polling information coming into his office. While presidents since then have varied in the extent to which they utilize such information – with Truman and Eisenhower using polls comparatively little and Carter, Reagan, and particularly Clinton using them more extensively – survey data have come to play an important role as the White House deliberates various policy options.

While other officials do not have the resources of the president, there are a number of ways in which they use survey data to take into account public opinion in developing policy or evaluating government practice. One such use involves surveys that are done as part of a needs assessment. In a needs assessment, an agency surveys its clients or potential clients to determine the services they are receiving and to identify those areas in which they feel new or additional services could be most useful. For example, an agency that provides aging services may survey senior citizens in the region it serves to determine their perceived needs in areas such as housing, transportation, meal services, recreation, housekeeping and maintenance, and respite services. Based on these results, the agency could then develop programs designed to address the greatest needs of the population. Virtually any type of government agency can make use of a needs assessment survey. For example, a school district could survey its teachers to determine what they perceived to be their greatest needs in terms of continuing education, instructional materials, classroom support, and community resources. Similarly, a recreation district could survey residents of an area to identify their perceived needs for facilities such as baseball fields, soccer fields, tennis courts, basketball courts, parks, scenic areas, walking and hiking trails, playgrounds, swimming pools, and the like. A substance abuse treatment agency could use survey information to identify the prevalence of use for different types of substances and to estimate the need for treatment services in the population as well as among various subgroups. In sum, needs assessment surveys can be employed in a variety of contexts to assist government agencies in utilizing their resources more effectively.

A related use of surveys by government organizations is as part of a program evaluation. While the purpose of needs assessments can be described as determining what programs should be developed and how services should be delivered, surveys conducted as part of a program evaluation are done to assess how effectively an agency is carrying out its functions. An agency involved with job placement, for example, could survey its clients to determine whether they had received training and how effective they felt the training was, the extent to which the services had helped them to find a job, whether they were still employed, their current salary, and other information related to their current job status. Likewise, a program designed to evaluate the effectiveness of underage drinking laws could conduct surveys of youth to determine their drinking-related behaviors and combine this with other measures in developing an overall assessment of the program's effectiveness. Any agency whose employees have contact with the public can use surveys to determine the extent to which the general public has contact with the organization, how such contact varies across subgroups, and evaluate various aspects of employee service, such as courteousness, knowledge, wait time, and overall satisfaction. Like needs assessments, surveys done as part of a program evaluation can be used for a variety of purposes and by virtually any type of government organization.

As noted in Anna Berger’s and Bill Tomes’ article earlier in this issue, another type of survey that is being used more frequently by governmental units is a “benchmarking” survey. As performance measurement in government has become the focus of increasing interest, the use of citizen surveys has become popular because they can provide more reliable and representative input from citizens than do other methods of gauging public opinion, such as direct contact, interest group lobbying, or letters to the editor. The idea behind benchmarking is that service performance cannot be adequately determined without some frame of reference. Comparisons can be made in one of three ways: (1) against some absolute standard; (2) against comparable agencies or municipalities (external benchmarking); or (3) against previous measures (internal or historical benchmarking). Benchmarking has a number of utilities for state and local government officials, in that it can provide valid and reliable indicators of how a state agency or local government is performing. Citizen (or customer) satisfaction and evaluation surveys are an important component of this process, in that attitudes about service delivery represent government's “bottom line.”  Benchmarking surveys at the municipal level can be used to assess a variety of services such as police, fire, recreation, garbage collection, road maintenance and the like. Although any benchmarking effort faces challenges in identifying appropriate external comparisons, monitoring citizen evaluations against an established standard and making cross-time comparisons of citizen satisfaction provide a valuable tool for public officials. And while the use of benchmarking surveys has increased in recent years, their value has long been recognized. Almost thirty years ago, for example, Webb and Hatry (1973) argued that "[Citizen surveys are] … possibly the most, if not the only, efficient way to obtain information on (1) constituents' satisfaction with the quality of specific services, including identification of problem areas; (2) facts such as the number and characteristics of users and nonusers of various services; (3) the reasons that specific services are disliked or not used, (4) potential demands for new services, and (5) citizen opinions on various community issues, including feelings of alienation toward government and officials."

An additional way in which government officials can use surveys is to assess the public's views on a specific policy issue. For example, a county considering the adoption of a local option sales tax might survey eligible voters to gauge support or opposition to such a tax and to identify the reasons underlying public sentiment, as well as any differences in opinion that might exist among subgroups. Similarly, a municipality that is considering alternative methods for the way it handles garbage collection might survey residents to determine which option citizens prefer, or a state agency faced with an option of increasing fees or reducing its services might survey the state's residents to determine the public's views on these alternatives.            

Public officials can also use survey data that are provided by other groups. Often when some policy decision is being considered at the state or local level groups that would be affected by the decision conduct their own polls in order to measure public support for their position. If the results show public support, these groups attempt to provide this information to policymakers either through the media, lobbying, or testimony before the group considering the policy options. For example, if the General Assembly were considering issues such as changes in the state's death penalty statute or permitting the medical use of marijuana, proponents on either side of these issues could commission a survey and provide lawmakers with the results demonstrating "the public's view."           

Overall, public officials have a wide variety of uses for data from public opinion polls. Such information can be used to determine how citizens feel about policy issues, to identify needs in areas such as aging, health, recreation and transportation, to monitor performance, to identify citizen budget priorities, and to evaluate specific policies, programs, or services.

CONDUCTING A QUALITY SURVEY            

Public officials involved in conducting a public opinion survey should understand that there are a number of steps involved in the survey process, from determining the population or group for which one is interested in obtaining information to analysis and reporting of data. While each of these steps is important, the key to a good survey is obtaining a representative sample.            

In a representative sample, each individual in the population in which you are interested should have some known probability of being selected. It is possible to collect information from a representative sample whether the data are collected by mail, in-person, by telephone, or by some electronic means, such as over the Internet. Each of these methods of data collection has advantages and disadvantages in terms of providing a representative sample.           

Collecting survey data face-to-face is generally considered to be the most effective method for obtaining a representative sample. This method, which involves having an interviewer contact the selected respondent and conducting the interview in the respondent's home, is used for a number of major surveys conducted by the federal government, such as the National Health Interview Survey. Virtually all citizens can be reached through an in-person survey, and an interviewer can make repeated attempts to locate the selected respondent and collect information. But the advantages of gathering survey data face-to-face have a cost: such surveys are much more expensive to conduct than those using methods such as mail or telephone. As a result, only a relatively small percentage of the surveys done in the United States today employ face-to-face data collection.            

In order for the results of a mail sample to be representative, there generally must be a list of members of the population from which a random sample can be selected to receive the survey. To the extent that any such list is incomplete or out-of-date, the ability to produce a representative sample is compromised. A greater threat to the representativeness of mail surveys is posed by the generally low response rates for such surveys. Even using the "best" procedures – multiple mailings of the questionnaire, reminder post-cards, a short questionnaire, and a well-designed cover letter that provides an inducement for the person to respond – such surveys generally achieve less than a 50% response rate, and in many cases this rate is closer to 20%. Low response rates increase the potential that the results will be systematically different from those in the population. While mail surveys have the advantage of being the least expensive to administer, they are typically not used when some general population such as residents of a city or the state are the population of interest, and are more frequently employed when some smaller, more easily identifiable group (for whom addresses are often available) is being studied.

Survey data collection by telephone has many of the advantages of face-to-face interviewing and typically produces response rates much higher than those of mail surveys. Close to 95% of the population of the United States can be reached by telephone. With the use of random-digit dialing, in which telephone numbers are randomly generated from all working area code and exchange combinations in an area, a sample can be selected in which all individuals living in households with telephones have a chance of being selected. This sampling procedure provides results which are representative of the population at about one-fifth the cost of a comparable face-to-face survey. Although telephone surveys have some limitations relative to face-to-face interviewing – for example, they generally must be shorter, visual materials cannot be presented, and the interviewer does not have visual cues in interacting with respondents – their great advantage in terms of cost makes them most appropriate for many studies of public opinion. While response rates for telephone surveys have been declining in recent years, the use of appropriate techniques for locating respondents and adjusting for non-response in the analysis of the data produces results from telephone surveys that are representative of the population.

In recent years survey researchers have begun to explore methods for using the Internet and other electronic means to collect survey information. While such techniques can be very effective when gathering data from a relatively limited population that has virtually universal Internet access, they have a number of limitations for collecting information from the general population. The most obvious of these is that of access, since only about 50% of the households in the United States have access to the Internet. A second obstacle involves selecting a representative sample, since a list of e-mail addresses from which a sample could be selected does not exist. Despite these limitations, organizations such as Harris Interactive and Knowledge Networks have conducted surveys in conjunction with elections in which the results of the surveys' trial heats have been close to the actual electoral outcomes. While much research remains to be done in this area, the future is likely to see the increased use of the Internet as a vehicle for collecting public opinion data.

Whatever the method of data collection – face-to-face, telephone, mail, Internet – the key to any sample is its representativeness. For in any survey it is not the opinions and characteristics of those from whom information is collected that are important, in and of themselves; rather it is the characteristics of the larger population whom these sampled individuals represent that are of concern to public officials. The systematic collection of information from a representative group of the public resulting from a properly designed public opinion poll provides policymakers with an accurate reflection of the public’s views.

Another important consideration in any survey involves the way that questions are worded. There is extensive research in survey methodology which shows that even minor variations in question wording can have an effect on survey results. The way in which questions are asked – whether they force people to select from choices or allow them to develop their own answers, the number of choices given to respondents, the tone of the words used, and the order in which the questions are presented – all can have an impact upon survey responses.

As an example of the effect that question wording can have, Tom Smith (1987) has shown that the percentage of the public who believed that the government was spending too little on "assistance to the poor" was consistently about 40% greater than the percentage who thought the government was spending too little for "people on welfare." The negative connotations associated with "welfare" eroded public support for public assistance. Likewise, the order in which questions are asked can affect the results. As Schuman and Presser (1996) have demonstrated, for example, a "general" question on the issue of abortion received support from 61% of the public when it was asked before a more specific abortion item, but was supported by only 48% when it came after the specific item. If policymakers were presented with the results of a survey in which the general question was asked first they would come to a different conclusion concerning the public's view on this issue than if given the results from a survey in which the general question was asked second. Such examples demonstrate that when presented with the results of a survey, one should look carefully at the questions to determine if they contain any potential biases and that they accurately reflect the characteristics, such as an opinions or attributes, in which a policymaker is interested. Among the things to be considered in evaluating survey questions are that they avoid "loaded" or inflammatory words, do not suggest that one alternative is preferable to another, are balanced in terms of the alternatives presented, do not use slang or colloquialisms, and are accurate in terms of any facts presented. A good survey question – particularly one dealing with a policy issue – will represent the options with which the policymaker or administrator is faced.

BEING A GOOD CONSUMER OF SURVEY RESEARCH            

While many public officials will not be involved in the design or execution of a public opinion poll, virtually all are "consumers" of survey research, whether it be reading a report of a survey from a government agency, a media poll, or the results of a survey conducted by an interest group. The previous section described the importance of selecting a representative sample and the wording of questions as key elements in designing a public opinion survey. In addition, there are a number of other factors that one should consider in evaluating poll results. [For a more extensive discussion of these factors, see Link and Oldendick (1997)].           

One additional consideration is the number of people from whom data are collected. While probability methods can be used to select a sample that is representative of the population, all samples – no matter how carefully drawn – are subject to sampling error. This does not imply that the data are wrong; rather, all samples are subject to such potential error due to the fact that data are not obtained from all members of the population. Sampling error is a function of the number of respondents – the larger the number of people from whom data are collected, the smaller the sampling error (and, of course, the higher the cost of the survey). A survey of 1,000 respondents would have a sampling error of +/- 3.1%, while in one with 200 respondents the sampling error would be +/- 6.9%. Knowing the sample size and the consequent probability for error due to chance is obviously an important consideration in evaluating survey results.           

Another element of public opinion polls to be aware of is that people will almost always give an interviewer a response to a question, even if they really don't understand or know anything about the topic. Because few people want to admit they are uninformed about issues that others might expect them to be informed about, survey participants will offer some type of opinion or response. Treating such "nonattitudes" as genuine expressions of public opinion can present an inaccurate picture of public attitudes. This is another reason why the wording of survey questions should be carefully examined. Does the question ask about a topic on which most respondents can be expected to have some information? Is the question written in a way that is easily understandable to most respondents? Is there an attempt to filter or screen out respondents who may not have a well-formed opinion on the issue?           

A similar concern about the accuracy of responses is raised by the fact that the survey interview is a social interaction between the interviewer and the respondent. As such, there is a tendency is some cases for respondents not to want to offend the interviewer and, therefore, to give a “socially desirable” response. For example, if a survey question deals with racial issues and the interviewer and respondent are of different races, the respondent may give an answer that reflects what they feel the interviewer wants to hear rather than expressing his or her true opinion. The potential that survey results reflect social desirability should be considered in interpreting the reported findings.

Consumers of survey information should also examine the data themselves in order to check the reported interpretation. While most survey results are reported honestly and objectively, it is always possible for a survey analyst to make a mistake in the way they present information or to allow their biases to affect the interpretation of the data. One should also keep in mind that what is not reported may be just as important as what is presented. Given the questions asked in a survey, are there results that are not reported or are there issues that the survey was designed to address that do not appear in the analyses? Carefully reviewing the survey and the analyses presented can often be useful in leading a consumer of survey research to ask “what’s missing?”           

A report of the results of a public opinion poll is a product. Good consumers will examine how the product was constructed – designed, executed, and analyzed – in deciding whether to “buy” the results.

CONCLUSION                

The results of public opinion surveys can be useful to government officials in a variety of ways, from their use in campaigns through identifying the public’s policy preferences and evaluating government performance. A well-designed opinion poll is a valuable tool in that it enables those entrusted with carrying out the public’s wishes to determine how those affected by some policy or program feel about an issue or evaluate the services they are receiving. As this article has tried to demonstrate, however, there are limitations to the use of surveys. A poorly designed survey – one that does not collect information from a representative sample or in which the questions are formulated in a way that biases the results – can be more harmful than useful in the policy process because it provides potentially misleading information that is not reflective of the public’s views, and may result in counterproductive policies. While citizen attitude surveys can play an important role in public policy and practice, they must be well-designed and executed in order to be of any value. The considerations described in this article should assist government officials at all levels in evaluating the utility of any survey results with which they are presented.

NOTES

¹At one point in time there was a greater distinction made between “polls” and “surveys.” Polls were generally shorter and dealt mainly with political or public policy issues, while surveys were somewhat broader, collected more contextual information, and were more concerned with explanation. Over the course of time, the distinction between these two terms has become blurred. In this article, these terms are used interchangeably.

REFERENCES

Link, Michael W. & Oldendick, Robert W. (1997). “’Good’ polls/’bad’ polls – How can you tell?: Ten tips for consumers of survey research.” South Carolina Policy Forum 8 (Fall): 12-23.

Schuman, Howard & Presser, Stanley (1996). Questions and answers in attitude surveys: experiments on question form, wording, and context. Newbury Park, CA: Sage Publications.

Webb, Kenneth & Hatry, Harry P. (1973). Obtaining citizen feedback: The application  ofcitizen surveys to local governments. Washington, DC: The Urban Institute.

ABOUT THE AUTHOR

Robert W. Oldendick, Ph.D., is the Executive Director of the University of South Carolina’s Institute for Public Service and Policy Research and a Professor of Government and International Studies. Dr. Oldendick has over 25 years experience in the field of survey research and public opinion. He is also the author of numerous publications on survey research methodology, including procedures for sample selection in telephone surveys, methods of respondent selection within households, and the effects of question wording and format on responses to survey questions. Together with Dr. Barbara Bardes, he is co-author of the text, Public Opinion: Measuring the American Mind, the second edition of which has recently been released by Wadsworth Publishing. Dr. Oldendick can be contacted at oldendick@iopa.sc.edu.