Polls, Polls, and More Polls An Evaluation of How Public Opinion Polls Are Reported in Newspapers Reed L. Welch This article examines the information the news media provide about public opinion polls.To do this, the study uses standards established by the American Association for Public Opinion Researchers and the National Council on Public Polls to evaluate how polls about the presidential election were reported in four major national newspapers and four smaller newspapers in the fall of 2000. The study found that some newspapers do better at providing information about polls than others,almost all articles do not disclose important information about polls, a newspaper does a better job at providing information about polls sponsored by the newspaper itself than it does about outside polls, and large, national newspapers do no better at reporting information about polls than smaller.more locally oriented newspapers.The article concludes that the media do not disclose the "minimal essential information" for the public to determine a poll's reliability and validity and that it would be in the best interest of polling organizations, newspapers,and the public for more information to be provided about the polls that are made public. Every four years the media gorge themselves on horse-race coverage of presidential campaigns (Asher 1992:273—78; Robinson and Sheehan 1983:146—48). Seemingly every day of a presidential campaign, poll results showing who is winning or losing, gaining or slipping are the subject of stories in newspapers and on television news. Even when poll results are not the subject of a story, they are frequently sprinkled into other stories and frame how stories are reported (Frankovic 1998; Hickman 1991:101; Jamieson 1993:40; Patterson 1994). "Public opinion polls have become staples of contemporary political reporting. Indeed, polls often seem to dominate coverage of campaigns and elections, producing incessant attention to the candidates' standing in the race and to their strategy and tactics" (MacLaury 1992:vii). According to Kathleen Frankovic Press/Politics 7(1): 102-114 © 2002 by the President and the Fellows of Harvard College 102 Welch / Reporting of Public Opinion Polls 103 (1998), director of surveys at CBS News, "Polls are not only part of the news today, they are news. They not only sample public opinion, they define it" (p. ISO). Because polls are an important part of newspaper and television coverage of campaigns, it is important to understand how the media report information about polls. The purpose of this article is to evaluate whether the media provide enough information when discussing polls for the public to determine the polls' reliability and validity. To do this, I examine how newspapers reported polls concerning the presidential election of 2000 from September 1 to November 7 (Election Day). Polling Industry Standards and Compliance Concern about the information given when poll results are reported is not new among polling organizations and journalists. The American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP) are the two most prominent sources that have proposed standards for polling organizations and the media to use when reporting poll results.1 In essence, they suggest that if certain pieces of information are not transmitted with the poll results, such results should be discounted and even ignored. The AAPOR (1999a) has stated that when public opinion researchers report results, they should include "certain minimal essential information about how the research was conducted to ensure that consumers of survey results have an adequate basis for judging the reliability and validity of the results reported." The NCPP (2000) has similar standards requiring that certain information is disclosed "to insure that consumers of survey results that enter the public domain have an adequate basis for judging the reliability and validity of the results reported." Both of these organizations have delineated what information should be disclosed when poll results are presented (e.g., the sponsor of the poll, the dates the poll was conducted, the sample size, the population that was sampled, the question's wording, etc.). There is considerable overlap in the guidelines set forth by each organization, but there are a few differences between them, highlighting the fact that there is no established dogma on what polling organizations and media need to reveal to the public about public opinion polls. Even with set guidelines, however, there is no guarantee that a polling organization or the media will follow these standards. The NCPP (2000) has said that polling organizations that report results are responsible for trying to have the media include information covering these guidelines in their news stories and should also "make a report containing these items available to the public upon request." Yet, despite any efforts the polling organization may make, "there really is no way to enforce this requirement with news organizations once the poll release has become a public document" (Asher 1998:90). As it stands now, 104 Press/Politics 7(1) Winter 2002 compliance with the standards is voluntary for both the polling organization and the medium that reports the results. Literature Review Studies have examined newspaper conformity to some of the AAPOR and NCPP guidelines. Miller and Hurd (1982) examined how closely the Chicago Tribune, the Los Angeles Times, and the Atlanta Constitution followed the guidelines set forth by the AAPOR. In a sample of 116 polls from 1972 to 1979, they found an increase in the number of polls reported but not an increase in conformity to AAPOR standards. They did find that conformity was higher when polls concerned election rather than nonelection topics and when the newspapers were the sponsors of the polls. Salwen (1985a, 1985b), using AAPOR standards, examined 264 polls stories published in the front section of both major Detroit newspapers from July 25 to the Monday before Election Day from 1968 to 1984. Salwen (1985a) found that the reporting of methodological information of the reported polls improved over time, and confirming what Miller and Hurd (1982) concluded, newspapers did a better job of reporting necessary methodological information with their own polls than they did with wire service and syndicated polls. Salwen (1985b) also discovered that the reporting of presidential polls did not improve the closer the election became, nor did presidential polls contain more methodological information than other polls. Another study examined how polls were reported in the Chicago Tribune from July 1, 1988, to December 31, 1989, a period that covered a presidential campaign and the primary and general election of the Chicago mayoral race (Brady and Orren 1992). The study found, similar to what Salwen (1985a) and Miller and Hurd (1982) found, that the Tribune did a better job of disclosing the necessary information of polls done for the Tribune than for outside polls. Nevertheless, even when polls were conducted for the Tribune, the articles frequently did not include essential information about the polls. All of these studies found that newspapers could do better at reporting information about polls than what they do; the papers usually reported some necessary information but, probably in part because of space restrictions, did not report other important information. These studies also found that newspapers did a much better job of reporting the details of in-house polls than they did with the results from wire service and syndicated polls. Data and Method In this article, I evaluate how polls about the presidential election were reported in newspapers from September 1, 2000, through Election Day, November 7, 2000, Welch / Reporting of Public Opinion Polls 105 using a combination of the guidelines established by the AAPOR and the NCPP. I examine four newspapers that are national in scope and are among the top five in the country in circulation (New York Times, Washington Post, USA Today, and the Wall Street Journal) and, unlike previous studies, examine four more locally oriented newspapers with much smaller circulations (Amarillo Daily News, the Daily Herald, the Santa Fe New Mexican, and the Wichita Falls Times and Record News). Thus, many more newspapers are included in this study than previous studies, allowing not only a comparison among large newspapers but also for the first time examining local newspapers and how they compare with large newspapers on their reporting of polls. The articles that were evaluated were stories in which polls about the presidential election and presidential candidates were discussed. Most of the stories were about who was ahead or behind in the polls, but there were others as well, such as stories showing the public's assessment of the candidates' characteristics, their leadership abilities, or who had the better understanding of policy. Other polls, such as estimating what voter turnout might be or who is leading in a Senate race, were not included. Stories were included in the evaluation even if the headline did not mention anything about polls and polls were not featured prominently in the article. In essence, most stories that mentioned polls were counted. On the other hand, stories were not included when polls were not referred to specifically but rather were used to set up the context of the race or were mentioned generically, such as "with Bush losing ground to Gore, he ..." or "most polls show the race is tied." Only regular newspaper articles were evaluated; editorials and columns on the Opinion pages were not included. Each of the articles was coded for whether or not it contained certain information about the polls. There were ten categories with which each article was evaluated. The ten categories were derived from the recommendations issued by the AAPOR (1986) and the NCPP (2000). After each of the ten recommendations, the organization that gives that recommendation is listed. 1. Sponsor of the poll (AAPOR, NCPP). 2. Sample size (AAPOR, NCPP). 3. Population that was sampled (AAPOR, NCPP). 4. A description of the sample selection procedure, giving a clear indication of the method by which the respondents were selected by the researcher, or whether respondents were entirely self-selected (AAPOR). 5. Date the poll was conducted (AAPOR, NCPP). 6. Question wording (AAPOR, NCPP). 7. Method of obtaining interviews (phone, in-person, etc.) (AAPOR, NCPP). 8. Sampling error (AAPOR). 9. Weighting or estimating procedures used (AAPOR). 10. Percentages on which conclusions are based (AAPOR, NCPP). 106 Press/Politics 7(1) Winter 2002 An article that contained information about more than one outside poll was counted only once and evaluated by how it most commonly reported information about the polls. For example, if an article mentioned three polls and gave the sample size to two of them, then the article was counted as providing the requisite information about sample size. If, on the other hand, the article had only listed the sample size to one of the polls, then the article was counted as not including the sample size since it had only given it for one of the three polls. There is a separate category for polls sponsored by a newspaper itself. If there were several polls mentioned in an article and one of them was an in-house poll, then the article was evaluated twice: once for the in-house poll and once for the other polls. Findings Table 1 shows how articles in the major newspapers in this study reported poll information. It shows that the newspapers were fairly consistent in the information they provided about polls; they consistently reported some information and also consistently did not provide other information. When examining all poll results (both the in-house polls and polls not sponsored by the paper), Table 1 shows that the papers did a good job of reporting both the sponsor of the poll and the actual poll results. In all four newspapers, these were clearly the two most reported items. Population was the third most reported item in all four newspapers, although this total was not reported as much as the sponsor and the poll results. The newspapers also frequently did not report other information. This was particularly the case with weighting, which only the New York Times brought up in articles discussing New York Times-sponsored polls. If all ten categories are totaled for all four newspapers, only twelve of the forty categories were covered in more than half of the articles in those newspapers. If sponsorship, poll results, and population are taken from the total, only one of the remaining twenty-eight categories had at least SO percent of the articles address that item. Table 1 clearly indicates what reporters and editors view as the most essential information to give about a poll: Who conducted or sponsored the poll? Whose opinion was asked? What are the results? These items tell the story reporters want to convey. If the main topic of a story is not about polls but rather uses polls to make a point or provide context to the story, reporters may be tempted to leave out some information about polls that the reporter may feel might distract the reader from the main point of the article. In the battle for paper space, editors may also ask why the method of gathering the information or the sampling error should be included in an article if the article is not about poll results but merely uses polls to provide context to the article. Table I Percentage of articles in four major newspapers reporting methodological information in poll stories Wall Street Journal New York Times Articles Articles Using Polls Using Polls Not Sponsored Articles Using All Poll Not Sponsored Articles Using All Poll Standard by Paper In-House Polls Articles by Paper In-House Polls Articles Poll sponsor 93 (26) 100 (9) 95 (35) 93 (37) 100 (12) 94 (49) Sample size 4 (1) 33 (3) 11 (4) 33 (13) 58 (7) 38 (20) Population 36 (10) 56 (5) 41 (15) 50 (20) 67 (8) 54 (28) Selection procedure 7 (2) 22 (2) 11 (4) 0 (0) 42 (5) 10 (5) Dates of poll 18 (5) 56 (5) 27 (10) 43 (17) 67 (8) 48 (25) Question wording 4 (1) 22 (2) 8 (3) 3 (1) 0 (0) 2 (1) Method of interviewing 0 (0) 22 (2) 5 (2) 0 (0) 42 (5) 10 (5) Sampling error 14 (4) 33 (3) 19 (7) 33 (13) 42 (5) 35 (18) Weighting 0 (0) 0 (0) 0 (0) 0 (0) 42 (5) 10 (5) Poll results 82 (23) 89 (8) 84 (31) 73 (29) 83 (10) 75 (39) Average for all ten categories (%) 26 47 31 33 54 38 Total number of articles 28 9 37 40 12 52 (continued) o Table I Continued Standard Articles Using Polls Not Sponsored by Paper Washington Tost Articles Using In-House Polls All Poll Articles Articles Using Polls Not Sponsored by Paper USA Today Articles Using In-House Polls All Poll Articles Poll sponsor 79 (23) 100 (32) 90 (55) 70 (28) 100 (69) 89 (97) Sample size 3 (1) 84 (27) 46 (28) 3 (1) 71 (49) 46 (50) Population 21 (6) 81 (26) 52 (32) 35 (14) 77 (53) 61 (67) Selection procedure 0 (0) 9 (3) 5 (3) 0 (0) 0 (0) 0 (0) Dates of poll 7 (2) 78 (25) 44 (27) 3 (1) 71 (49) 46 (50) Question wording 0 (0) 66 (21) 34 (21) 0 (0) 3 (2) 2 (2) Method of interviewing 0 (0) 22 (7) 11 (7) 3 (1) 17 (12) 12 (13) Sampling error 3 (1) 66 (21) 36 (22) 15 (6) 74 (51) 52 (57) Weighting 0 (0) 0 (0) 0 (0) 0 (0) 0 (0) 0 (0) Poll results 93 (27) 97 (31) 95 (58) 83 (33) 96 (66) 91 (99) Average for all ten categories (%) 21 60 41 21 51 40 Total number of articles 29 32 6 1 40 69 109 Note: Number of articles appear in parentheses. Welch / Reporting of Public Opinion Polls 109 Combining the in-house and outside polls together hides the considerable difference in the information reported about these two kinds of polls. Consistent with the findings in previous studies (Brady and Orren 1992; Miller and Hurd 1982; Salwen 1985a), the major newspapers in this study provided more information about polls sponsored by the individual newspapers than they did for outside polls. If all ten categories are combined for all four newspapers, Table 1 shows that only nine of the categories were discussed in half of the articles discussing outside polls, with eight of them being the poll sponsor and poll results categories. In contrast, more than half of the articles that discussed in-house polls addressed at least half of the categories in the articles (twenty-two out of forty). When the newspaper spends its own money to sponsor a poll and has its name attached to the poll, the paper highlights the poll and provides more information about the poll to build the poll's credibility and the prestige of the newspaper. Although all four newspapers fulfilled more of the AAPOR and NCPP requirements with their own polls than for outside polls, there are differences among the newspapers, however, in how they reported information about in-house and independently conducted polls. There was not as much difference in the Wall Street Journal between the two kinds of polls as there was in the other newspapers. Rather than indicating that the Wall Street Journal did a good job of reporting on outside polls, it instead shows that the Wall Street Journal did not provide as much information about its own polls as the other newspapers did about theirs. Indeed, Table 1 shows that the Wall Street Journal did not report information about polls as well as the other newspapers. The biggest discrepancy in the information newspapers provided about in-house and outside polls is seen with the Washington Post, which covered its own polls much better than any other newspaper did but was the worst in its reporting of outside polls. There was also a difference among these papers in the number of articles that even used polls. The Wall Street Journal reported fewer polls than the other major papers. This is not surprising, though, given that the Wall Street Journal does not publish on the weekends and does not dedicate as much space as the other three newspapers to national politics. On the other end of the spectrum, USA Today, which also only publishes on weekdays, used polls far more often than the other newspapers. USA Today seemed more focused on the horse-race aspects of the campaign than the other papers. Beginning on September 8, USA Today printed the results of daily tracking polls conducted by Gallup. Thus, every weekday USA Today had at least one story, however small it might be, that highlighted the latest tracking poll results and frequently used their tracking polls as well as other polls for additional stories. Just as there are differences among the four large newspapers in this study in the number of articles that included polls in the story, there is also a difference between large newspapers and the smaller newspapers in the number of stories that included polls (see Tables 1 and 2). The nationally oriented newspapers had 110 Press/Politics 7(1) Winter 2002 Table 2 Percentage of articles in four local newspapers reporting methodological information in poll stories Amarillo The New Times and Sponsor Daily News Daily Herald Mexican Record News Poll sponsor 41 (7) 75 (6) 79 (15) 60 (6) Sample size 12 (2) 38 (3) 16 (3) 40 (4) Population 35 (6) 38 (3) 53 (10) 30 (3) Selection procedure 0 (0) 0 (0) 0 (0) 0 (0) Dates of poll 24 (4) 38 (3) 21 (4) 60 (6) Question wording 12 (2) 13 (1) 0 (0) 0 (0) Method of interviewing 0 (0) 0 (0) 11 (2) 0 (0) Sampling error 18 (3) 38 (3) 16 (3) 30 (3) Weighting 0 (0) 0 (0) 0 (0) 0 (0) Poll results 59 (10) 63 (5) 84 (16) 80 (8) Average for all ten categories (%) 20 30 28 30 Total number of articles 17 8 19 10 Note: Number of articles appear in parentheses. many more articles using poll results than did the local newspapers. Not surprisingly, the national newspapers, with many more pages allotted to national news, had more space for more stories about the presidential race, which resulted in more stories that included polls. What is surprising is that there is not much difference between national newspapers and local newspapers in the quality of their coverage of polls. One would expect that the national newspapers would not only have more space to provide information about polls but that the reporters and editors of national newspapers would be more cognizant of information that should be included when discussing polls and would consequently do better at providing information about polls than local newspapers. This was not the case, however. If the reporting of outside polls in large newspapers is compared to the coverage in the smaller newspapers, the reporting was not very different. Indeed, in some cases the local newspapers did a better job of giving necessary information in their articles. Like the major newspapers, the smaller newspapers reported the sponsor of the polls and the results most often, and like the major papers, did not disclose much of the information recommended by the AAPOR and the NCPP. Discussion Did newspapers do a good job of reporting basic information about public opinion polls during the presidential campaign of 2000? Using the standards set by the AAPOR and NCPP, newspapers in this study almost always did not Welch / Reporting of Public Opinion Polls 111 provide the "minimal essential information" about the polls referred to in their articles (AAPOR 1999a). The two items about polls that newspapers consistently reported were the sponsor of the poll and the actual poll results. Other important information that the AAPOR and the NCPP recommend, however, was virtually ignored. An example from the fall 2000 campaign season highlights why newspapers should include more information than just the sponsor of a poll, the results, and the population sampled. USA Today, beginning September 8, printed the results of daily tracking polls conducted by Gallup. The results from these tracking polls received widespread attention not only because Gallup is a big name but also because they sometimes fluctuated erratically from one day to the next, leading many to question the results and Gallup's methods. There were reasons why Gallup's results fluctuated and differed from the results from other polling organizations, yet if people read the USA Today or other newspaper articles discussing polls, they would not have been aware of these reasons. Although a few columnists explained why Gallup's results differed from other polls, readers of USA Today and other newspapers could not have reached this conclusion on their own because the newspapers did not disclose the necessary information about the Gallup tracking polls or any other poll. If newspapers had followed the AAPOR and NCPP recommendations, however, readers would not have had to depend on columnists to find out why Gallup's results were different from other polling organizations but rather could have potentially determined this on their own and decided the validity and reliability of the polls (which is the purpose of the AAPOR and NCPP guidelines). There are few hard and fast rules of how a polling organization should conduct its polls. John Zogby of Zogby International said, "Twenty percent of this business is art, 80 percent is science." Each polling organization has its own way of doing things. That is fine as long as the public is aware of what polling organizations are doing. Yet, the readers of newspaper articles are not privy to much of the information that affects the results a polling organization receives. Weighting, for example, with the exception of the New York Times reporting on its own polls, was never reported in articles even though the weighting of such variables as sex, race, or party identification could change the raw numbers from a Bush lead to a Gore lead. The newspapers in this study appear to be aware of the importance of at least some of these standards as evidenced by their providing much better information about their own polls than for outside polls. Nevertheless, reporters and editors do not necessarily agree that information listed by the AAPOR and NCPP is important enough to be included in articles in which polls are discussed. Newspapers at times do not feel they warrant taking up precious newspaper space. If poll results are not the main point of the article, reporters and editors may feel that giving more information than the sponsor of the poll and the poll results 112 Press/Politics 7( 1) Winter 2002 would detract from the point of the story. Moreover, reporters and editors may also assume—probably correctly—that most readers do not want to sort through the minutiae of how a poll was conducted. Reporting polls in a cavalier fashion is shortsighted, however. It is in the best interest of both newspapers and polling organizations to have newspapers disclose more information about polls than they are currently doing. Perhaps part of the public's distrust, confusion, and skepticism about polls can be traced in part to polling organizations and the media not disclosing the information the public needs to know about polls. With the news media, and perhaps even polling organizations, not heeding the recommendations by the AAPOR and the NCPP, plenty of ammunition is provided to critics, skeptics, and detractors of polling. "What are they trying to hide?" they ask. Reputable polling organizations do not have anything to hide. They should want information about their polls in the papers. Their reputation and business is on the line. Yet, by revealing little information about the polls, suspicion is cast and credible organizations are lumped together with fly-by-night operators. Newspapers too should want to provide information about polls. With newspapers relying so heavily on polls, it is important that the public has some level of confidence in public opinion polls and not feel that the papers and polling organizations are trying to put one over on them. Providing the information that the AAPOR and NCPP recommend would lead to greater public understanding of polls and to an increase in the public's confidence in them. Given the limited information that usually accompanies polls in newspaper articles, however, the public should be distrustful of polls. The typical article does not give information on the sample size, when the poll was conducted, methods, question wording, and other "minimal essential information" (AAPOR 1999a). Indeed, given the limited information that newspapers usually provide about polls, a wise reader should dismiss the polls altogether. Acknowledgment The author would like to thank Theo Edwin Maloy for his helpful comments on the article. Notes 1. The American Association for Public Opinion Research (AAPOR) was founded in 1947 by public opinion researchers with the purpose of establishing professional standards and advancing research theory and methodology (American Association for Public Opinion Research 1999b). The National Council on Public Polls (NCPP) was created in 1969 and is an association of more than forty polling organizations. Its mission is to set standards for public opinion pollsters and to help politicians, media, and the public to understand how polls are conducted and how to interpret them (National Council on Public Polls 2001). Welch / Reporting of Public Opinion Polls 113 2. Richard Morin, "Unconventional Wisdom," Washington Post, Oct. IS, 2000:sec. B; and Joseph Perkins, "Tilted Political Polls," Washington Times, Nov. 4, 2000:sec. A. 3. Dana Milbank, "I Hear America Ringing; and Then Hanging up, More Likely than Not, If You're a Pollster during Election Season," Washington Post, Oct. 16, 2000:sec. C. References American Association for Public Opinion Research. 1986. "Ethics Code" retrieved Sept. 9, 2000: http://www.aapor.org/ethics/code/html American Association for Public Opinion Research. 1999a. "Best Practices for Survey and Public Opinion Research,"retrieved Sept. 9, 2000: http://aapor.org/ethics/best.html American Association for Public Opinion Research. 1999b. "History of AAPOR," retrieved Sept. 9, 2000: http://www.aapor.org/about/history.html Asher, Herbert. 1992. Presidential Elections and American Politics. 5th Edition. Pacific Grove, CA: Brooks/ Cole. Asher, Herbert. 1998. Polling and the Public. 4th Edition. Washington, D.C.: CQ Press. Brady, Henry E., and Gary R. Orren. 1992. "Polling Pitfalls: Sources of Error in Public Opinion Surveys." In Media Polls in American Politics, ed. Thomas E. Mann and Gary R. Orren. Washington, D.C.: The Brookings Institution. Frankovic, Kathleen A. 1998. "Public Opinion and Polling." In The Politics of News: The News of Politics, ed. Doris Graber, Denis McQuail, and Pippa Norris. Washington, D.C.: CQ Press. Hickman, Harrison. 1991. "Public Polls and Election Participants ."In Polling an d Presidential Election Coverage, ed. Paul J. Lavrakas and Jack K. Holley. Newbury Park, CA: Sage. Jamieson, Kathleen Hall. 1993. "The Subversive Effects of a Focus on Strategy in News Coverage of Presidential Campaigns." In 1 -800-President: The Report of the Twentieth Century Tund Task Torce on Television and the Campaign of 1992. New York: The Twentieth Century Fund. MacLaury, Bruce K. 1992. "Foreword." In Media Polls in American Politics, ed. Thomas E. Mann and Gary R. Orren. Washington, D.C.: The Brookings Institution. Miller, M. Mark, and Robert Hurd. 1982. "Conformity to AAPOR Standards in Newspaper Reporting of Public Opinion Polls." Public Opinion Quarterly 46(summer):243—49. National Council on Public Polls. Retrieved Mar. 23,2001: http: / / www.ncpp.org/home.html National Council on Public Polls. "Statement of Disclosure," retrieved Sept. 5, 2000: http:// www. ncpp. org. disclo sure. html Patterson, Thomas E. 1994. Out of Order. New York: Vintage. Robinson, Michael, and Margaret Sheehan. 1983. Overthe Wireandon TV. New York: Russell Sage Foundation. Salwen, Michael B. 1985a. "Does Poll Coverage Improve as Presidential Vote Nears?"Journalism Quarterly 62(winter):887-91. Salwen, Michael B. 1985b. "The Reporting of Public Opinion Polls during Presidential Years, 1968-1984rJournalism Quarterly 62(summer): 272--77'. Biographical Note Reed L. Welch is an assistant professor of political science at West Texas A&M University. He has published articles in journals such as the American Journal of Political Science, Congress and the Presidency, Communication Annual, and the Journal of Leadership Studies. 114 Press/Politics 7(1) Winter 2002 Address: Department of History and Political Science, West Texas A&M University, WTAMU Box 60807, Canyon, TX 79016-0001; phone: 806-651-2431; fax: 806-651-2601; e-mail: rlwelch@mail. wtamu. edu. Paper submitted May 10, 2001; accepted for publication August 22, 2001.