H) Check for updates Article American Politics Research 1-19 ©The Author(s) 2019 Article reuse guidelines: sagepub.com/journals-permissions DOI: 10.1177/1532673X19840732 journals.sagepub.com/home/apr (DSAGE Christopher B. Mann1©, Kevin Arceneaux2, and David W. Nickerson3 Abstract A multitude of laboratory experiments show that subtle shifts in framing can induce individuals to participate in political activity. Using four randomized field experiments, we tested whether exposure to messages framing public policy proposals negatively increased political action relative to exposure to messages framing the proposal positively. Three experiments use a type of political participation novel to the field experiments literature: phone calls recruiting people to contact elected officials. Contrary to expectations from prior laboratory experiments on intention to participate in collective action in politics, we find scant evidence that messages framed negatively about the policy returns from participation are more effective than messages framed positively about the policy returns from participation at motivating real-world political behavior. Keywords political behavior, framing, negativity bias, field experiment, policy returns to participation 'Skidmore College, Saratoga Springs, NY, USA 2Temple University, Philadelphia, PA, USA 3University of Notre Dame, IN, USA Corresponding Author: Christopher B. Mann, Skidmore College, Saratoga Springs, NY 12866, USA. Email: cmann@skidmore.edu Do Negatively Framed Messages Motivate Political Participation? Evidence From Four Field Experiments 2 American Politics Research 00(0) The vast majority of citizens' only opportunities to participate in the political process are through acts such as voting in regular elections and communicating opinions to representatives (e.g., contacting elected officials, attending community meetings, engaging in public protest, etc.) and many field experiments demonstrate that personal blandishments by a stranger can induce some individuals to participate (e.g., Green & Gerber, 2015). Civic and political organizations seeking to increase these forms of participation frequently frame the reasons to participate negatively (i.e., harm, damage, threat, loss) or positively (i.e., benefits, improvement, aspiration, gain) about salient public policy outcomes. Although there is considerable evidence from laboratory experiments in political science and other fields that negative frames should produce larger effects, we are not aware of any prior research that compares negative versus positive frames in field experiments about policy returns from real-world participation in politics. In this article, we evaluate whether employing negative or positive frames to describe policy proposals better motivates citizens to engage in political behavior in the real-world collective action settings of voting and contacting their elected representatives. Contrary to expectations from laboratory experiments, we find little-to-no evidence supporting the hypothesis that negatively framed messages about policy returns will increase real-world participation in collective action in politics for the overall population or any subgroup. Although some prior field experiments comparing frames for encouraging real-world political behavior have found few differences between competing frames (Arceneaux & Nickerson, 2010; Bhatti, Dahlgaard, Hansen, & Hansen, 2018; Gerber, Huber, Fang, & Reardon, 2018; Green and Gerber, 2015; Mann, 2010; Nickerson, 2007), the failure to find successful frames might be due to these previous comparisons using weak and atheoretical distinctions between the treatments. Field experiment manipulations more firmly rooted in behavioral theory have found that concepts such as accountability (Gerber, Green, & Larimer, 2008; Panagopoulos, 2011a), identity (Valenzuela & Michelson, 2016), and implementation intentions (Nickerson & Rogers, 2010) can motivate real-world behavior better than typical appeals. Several streams of research in economics, psychology, and political science provide a firm theoretical foundation for the prediction that negative frames should motivate political behavior more than positive frames. First, people have a general predisposition to privilege negative information over positive information (Hibbing, Smith, & Alford, 2014), and as a result, emphasizing the negative aspects of policy proposals tends have a stronger and more lasting effect on policy opinions than emphasizing the positive (Cobb & Kuklinski, 1997). Second, people's tendency for "losses [to] loom Mann et al. 3 larger than gains" (Kahneman & Tversky, 1979, p. 279) causes people to prefer policies that avoid losses more than policies that promote gains (e.g., Arceneaux, 2012; Druckman, 2004; Tversky & Kahneman, 1981). Third, people's aversion to losses coupled with their tendency to value the things that they have more than the things that they do not (the "endowment effect"; see Kahneman, Knetsch, & Thaler, 1990) leads many people to resist supporting proposals that shift away from the status quo (e.g., Quattrone & Tversky, 1988). Many studies shows that negativity bias, loss aversion, and status quo bias shape political attitudes. Although it is tempting to draw a causal arrow from attitudes to behavior, the intention to do something—which is an attitude— tends to be weakly correlated with actual behavior (Ajzen, 1991). In fact, the evidence that negative information motivates political behavior more than positive information is mixed. For instance, negative campaign advertising does not clearly motivate voter turnout, as dozens of studies find everything from a demobilization effect to a mobilization effect (Lau, Sigelman, & Rovner, 2007). Similarly, Miller and Krosnick (2004) found in a large-scale field experiment that people were more likely to contact the president if a change from the status quo was framed positively (opportunity) rather than negatively (threat), which is contrary to expectations. Recent field experiments on voter mobilization also find little evidence of difference when using negative and positive frames of several aspects of voting. Mobilization treatments using positive descriptive norms (i.e., joining others in voting) or negative descriptive norms (i.e., not joining others who fail to vote) do not have different effects on turnout (Gerber et al., 2018). Treatments encouraging voting with negative frames of political efficacy ("do not let others decide") or positive frames of political efficacy ("take part in deciding") do not produce different effects on voting (Bhatti et al., 2018). Evoking explicit social pressure with a negative frame ("shame") appears more effective at mobilizing voter participation than using positively framed social pressure ("pride"), but the difference is conditional the type of recipient (Panagopoulos, 2010). Thus, while the link between negativity bias, loss aversion, and status quo bias and political attitudes appears strong, the effect on real-world political behaviors requires more testing. There is more consistent evidence that loss aversion motivates nonpoliti-cal behavior. Across a multitude of laboratory experiments, individuals are less likely to take from a common pool than they are to contribute to a public good, even though these are essentially identical games (e.g., Brewer & Kramer, 1986; De Dreu & McCusker, 1997; Krishnamurthya, Bottomb, & Rao, 2003; McCusker & Carnevale, 1995; Sell & Son, 1997; Van Dijk & Wilke, 2000). Within the domain of public health, loss framed messages are 4 American Politics Research 00(0) more successful at encouraging people to engage in healthy behaviors than gain framed messages are (e.g., Levin, Schneider, & Gaeth, 1998; Meyerowitz & Chaiken, 1987). These studies demonstrate that loss aversion can motivate behavior in laboratory settings among subjects playing abstract economic games as well as motivating health behaviors that provide a direct benefit to the message recipient, but it is less clear that these findings extend to real-world settings where people bear personal costs for contributing to a public good through political behavior. We conducted four field experiments to test the power of negatively framed messages about policy returns to participation to motivate political behavior in real-world collective action settings. Our negative frames combined general negative information, loss frames about policy returns, and argued for the status quo. This approach should make the negative frame as powerful as possible from a theoretical standpoint, as well as reflect real-world political messaging strategies. After an initial voter mobilization experiment (a staple of the political science field experiments literature) found no support for the hypothesis negatively framed messages about policy returns are more effective at increasing participation in real-world politics than positively framed messages, we conduct three field experiments in a novel context: phone calls by a policy advocacy organization to recruit and facilitate contacting an elected official. Contacting elected officials is an important real-world collective action setting for study and is included in many common scales of political engagement, but these experiments are the first of their kind. Methodologically, these experiments allow us to eliminate the possibility that the decay of induced treatment effects during the relatively long duration between treatment and voting accounts for the initial null findings. Contacting elected officials is a meaningful, real-world political participation that can be undertaken immediately after treatment (i.e., opting to have the call connected to the elected official's office). Substantively, these types of calls are frequently used in lobbying efforts, and our treatments reflect framing often used about policy returns to participation. These field experiments found little evidence that negative frames about policy returns to participation motivate political action more than positive frames for the overall population or any subgroup. Considering the invidious threat of publication bias to the scientific enterprise (Rosenthal, 1979), it is crucial for scholars to catalog null results that arise from powerful designs, especially when applying laboratory findings to field settings. The conclusion discusses possible explanations for the null findings to spur future research and contextualize these results. Mann et al. 5 Table I. Field Experiment I Script Manipulation. Positive frame There are a lot of candidates and issues on the ballot this year, and each of them is important for our future. It takes all of us to get involved so that we can improve the economy and enjoy clean air and clean water. We're asking people to pledge to fill out their entire ballot. Can we count on you to try to fill out the entire ballot? Negative frame There are a lot of candidates and issues on the ballot this year, and each of them is important for our future. It takes all of us to get involved so that we can avoid job loss and protect clean air and clean water. We're asking people to pledge to fill out their entire ballot. Can we count on you to try to fill out the entire ballot? Note. Emphasis added. Full script is available in the Supplemental Materials. Field Experiment I: Voter Mobilization Design Live phone calls encouraging turnout among registered voters in Illinois, Michigan, New York, and Pennsylvania were administered during the weekend prior to Election Day for the 2010 general election.1 Subjects were randomly assigned to one of the four experimental conditions, only two of which are relevant to our comparison of frames (Table 1): (a) a positively framed version of the phone script including language about prospective gain in policy outcomes from voting («assigned = 25,181, «contacted = 8,698); or (b) a negatively framed version of the phone script including language about prospective loss in policy outcomes from not voting («assigned = 25,214, «contacted = 8,419). We focus on differences in turnout among the people contacted in each condition because the frame can influence behavior only when it is actually delivered.2 Subjects assigned to the positive and negative frames were balanced across observable covariates as expected (see Supplemental Table SI for details). A manipulation check conducted with an independent sample provides evidence that people were more likely to describe the negative frame using negative adjectives than they were when describing the positive frame (p = .013, see Supplemental Materials for details). These types of paid phone calls are a staple of voter mobilization efforts. Previous field experiments show that live phone calls with a conversational tone can significantly increase voter participation (e.g., Ha & Karlan, 2009; Mann & Klofstad, 2015; Nickerson, 2006, 2007). This experiment has a high degree of realism because it was conducted in 6 American Politics Research 00(0) Table 2. Field Experiment I: Turnout in Positive and Negative Conditions Among Contacted Registered Voters. n (contacted) Turnout SE Positive Script 8,698 37.2% 0.5 Negative Script 8,419 38.0% 0.5 Difference (percentage points) 0.8 0.7 partnership with a nonpartisan charitable organization as part of its planned 2010 voter mobilization program.3 Results After the election, we obtained public records on individual-level voter turnout to ascertain whether subjects voted in the election. Among all records assigned to an experimental condition, both the positive and negative frame treatments appear to increase turnout relative to the control group, although this intent-to-treat effect is only statistically significant for the negative frame treatment.4 The influence of the framing is measured by comparing turnout among people to whom each randomly assigned treatment was delivered (i.e., people successfully contacted by phone). Despite the considerable statistical power of Field Experiment 1 relative to similar laboratory experiments, Table 2 reports the increase in turnout from the negative frame treatment was not statistically distinguishable from the positive frame treatment (0.8 percentage points, SE = 0.7, p = .25). This difference may appear substantively noteworthy in the context of modest effects from phone call mobilization treatments (for a meta-analysis, see Green & Gerber, 2015), but the uncertainty requires treating the result of this single experiment as no more than very weak evidence—and more properly as a null result. The null difference between the treatments is not simply an artifact of null overall effect, as both treatments appear to have increased turnout relative to the control group. Field Experiments 2 to 4: Patch-Through Policy Advocacy Calls Design Although the treatments in Field Experiment 1 may have failed to reveal large framing effects for several reasons, one important possibility is the Mann et al. 7 time lag between treatment and subject's ability to carry out the behavior. Laboratory experiments on loss aversion and collective action generally measure prosocial behavior immediately after the stimulus, but framing effects are short-lived even in laboratory settings and especially when the frame is not repeated (e.g., Chong & Druckman, 2010). Kalla and Broockman's meta-analysis of campaign persuasion field experiments reports that political framing decays close to Election Day (Kalla & Broockman, 2018). Experiment l's failure to find evidence that negatively framed messages are more effective than positively framed messages could simply reflect decay in framing effects.5 Therefore, we sought to conduct additional field experiments where real-world political participation occurs immediately after the treatment stimulus. Field Experiments 2 to 4 solve the time lag problem by using live calls recruiting citizens to contact their governor about the governor's stances on proposed changes to environmental regulations. These "patch-through" phone calls offer a valuable way to investigate political behavior, and we are not aware of published research utilizing them. The outcome of interest is agreeing to have the call transferred to the elected official's office ("patched-through"), which is a meaningful measure of political behavior and an act taken immediately after stimulus delivery (like laboratory experiments). The phone calls recruiting volunteers to be patched through allow flexibility in crafting stimuli and monitoring of call delivery. Patch-through calls are a common tactic of policy advocacy organizations to generate "grassroots" input into the policy process. Field Experiments 2 to 4 were part of a larger advocacy effort by our partner organization. Experiments 2 to 4 are linked not only by a common behavior to be studied, but also a common state and subject pool. In early December 2011, our partner organization defined a list of 60,619 households that were deemed likely to support its policy positions (see Supplemental Materials for details). Each targeted household was randomly assigned to receive a positive or a negative frame about policy returns in the recruitment call and the entire list of 60,619 households was placed in a random order. Randomizing well in advance of the program does not compromise the internal validity of the experiment and was a practical necessity because our partner organization anticipated the Governor's office would make policy decisions on short notice and randomizing files in real time would introduce delays. When our partner organization felt calls from supporters to the governor were needed, it determined how many patch through calls they could accommodate and uploaded the first households from the randomized list up to the number necessary. After the action period was over, these households uploaded into the system were deleted from the list and would not be 8 American Politics Research 00(0) called again. Placing households in a random order allowed the organization to include as many households as it needed, guaranteed that each household was included in only one experiment during this period, and ensures that the same types of households were recruited to call the governor for each action.6 Positive and negative scripts were written for each of the patch through experiments and applied to each household according to the initial random assignment of script type. The random assignment to one of the two conditions was balanced across observable covariates in each experiment and in the overall population as expected (see Supplemental Table S2 for details). Field Experiment 2 (n = 13,439) was conducted in December, 2011, using scripts about the governor's proposed change to a state administrative rule that our partner organization expected to lead to increased carbon pollution in the state (Table 2). The experiment randomly assigned subjects to one of the two conditions defined by variations in the phone script: (a) positive frame focusing on the prospective policy gains from keeping the existing pollution rule («assigned = 6,707, «contacted = 518) or (b) negative frame focusing on the prospective policy losses from the proposed change to the rule («assigned = 6,732, «contacted = 522). We recorded whether subjects chose to "patch through" to the governor's office after listening to the script. In this field experiment (as well as Experiments 3 and 4), our partner organization sought to retain the status quo policy. Therefore, the negative and positive frames are asymmetric if there is bias toward retaining the status quo policy. Thus, our treatment design intentionally fuses negativity bias, loss aversion, and status quo bias, which should make the negative frame as powerful as possible from a theoretical standpoint. A manipulation check conducted with an independent sample provides evidence that people were more likely to describe the negative frame using negative adjectives than they were when describing the positive frame (p < .0001, see Supplemental Materials for details). Field Experiments 3 and 4 targeted the governor's proposed changes to an administrative rule (known as "the pit rule") requiring oil and gas production companies to dispose of toxic waste properly and prevent groundwater contamination (Table 3). Field Experiment 3 was conducted in January, 2012 (n = 13,781), before the state's first rule-making meeting. A postponement in the rule-making process allowed replication with the same scripts in April, 2012 (n = 33,399). As in Field Experiment 2, subjects were randomly assigned to one of the two conditions defined by variation in the phone script: (a) positive frame emphasizing the prospective policy gains from keeping the rule (Field Experiment 3: «assigned = 6,941, Mann et al. 9 Table 3. Field Experiment 2 Script. Positive frame Last year, adopted a strong rule that reduces carbon pollution by the biggest polluters in the state . . . Unfortunately, Governor wants to overturn the rule that reduces carbon pollution in . .. By keeping the rule, we can create good-paying jobs in the clean energy sector—at a time when we desperately need them. We'll also improve our air quality and become a national leader in tackling climate change. Negative frame Last year, adopted a strong rule that reduces carbon pollution by the biggest polluters in the state . .. Unfortunately, Governor wants to overturn the rule that reduces carbon pollution in . .. If the rule is dismantled, we will lose the good-paying jobs in the clean energy sector—at a time when we desperately need them. We'll also make the threats of climate change worse—including greater risks of wildfires and drought. Note. Emphasis added. Full script is available in the Supplemental Materials. "contacted = 494; Field Experiment 4: «assigned = 16,722, «contacted = 1,025) or (b) negative frame emphasizing prospective policy losses from losing the rule (Field Experiment 3: «assigned = 6,840, «contacted = 480; Field Experiment 4: «assigned = 16,677, «contacted = 1,032). These studies build on Field Experiment 2 by peppering the negative or positive language throughout the script, creating a more powerful manipulation. We recorded whether subjects chose to "patch through" to the governor's office after listening to the script delivered by live professional callers. A manipulation check conducted with an independent sample provides evidence that people were more likely to describe the negative frame using negative adjectives than they were when describing the positive frame (p < .0001, see Supplemental Materials for details). Results Table 4 reports the results of Field Experiments 2 to 4. An interesting descriptive outcome is high percentage of subjects who agreed to be patched through to their representative and leave a message. In each of the three experiments, more than 20% of the subjects took the unusual step of making their views known to elected representatives. Admittedly, the subjects were modeled to support environmental causes and not a representative sample of the electorate, but these are not people who have necessarily taken prior actions. Our treatment reduces the logistical difficulties to calling representatives and applies 10 American Politics Research 00(0) Table 4. Field Experiments 3 and 4 Script Manipulation. Positive frame N With a strong pit rule, we can make sure our water is clean and safe. If Governor hears from enough people, she'll think twice about trying to dismantle the rule. She can request that her Oil Conservation Commission keep the pit rule, and make it stronger. We can patch you through to Governor 's office right now. All you have to do is tell her staff that you want her to protect our water with a strong rule for oil and gas waste pits. Can we patch you through to her office right now? egative frame Without the pit rule, our water is at risk of irreversible contamination. If Governor hears from enough people, she'll think twice about trying to dismantle the rule. She can request her Oil Conservation Commission not to weaken or get rid of the rule. We can patch you through to Governor 's office right now. All you have to do is tell her staff that you don't want toxic waste contaminating our water, so you want a strong rule for oil and gas waste pits. Can we patch you through to her office right now? Note. Emphasis added. Full script is available in the Supplemental Materials. mild social pressure to take action, but does nothing to ameliorate the awkwardness of confronting public officials. Thus, this relatively high rate of participation in our dependent variable suggests that fear of speaking out on a topic is not an insurmountable hurdle to contacting elected officials. Turning our attention to the estimated treatment effect from our three patch-through experiments (Table 5), only Field Experiment 3 found statistically significant evidence supporting the negative frame hypothesis (b = 5.8, SE = 2.7, p = .032) despite the large and statistically significant effect sizes commonly found in laboratory experiments on negative framing. However, even this result failed to replicate in Field Experiment 4 with the exact same treatments in an identical population and a sample twice as large. The pooled effect from Field Experiments 3 and 4 is far from statistically significant (b = 2.7, SE = 2.7, p = .499) and can definitively rule out framing effect sizes larger than 8 percentage points. The noisy pattern of results across the Experiments 2 to 4 (one with a negative sign, one with a positive sign, and one near zero) bolsters the inference that we are not drawing from a distribution centered around a large average effect. Three experiments cannot be conclusive, and it is possible that unseen factors account for the variation in results, but many explanations can be ruled out. For instance, the substantive consequences of the policies differed across the negative and positive frames in Field Experiment 2, and it is possible that these differences drove the negative Table 5. Field Experiment 2 to 4: Patch-Through Rate in Positive and Negative Conditions Among Contacted Registered Voters. Field Experiment 2: Field Experiment 3: Pit rule Field Experiment 4: Pit rule Pollution patch through patch through (January) patch through (April) n Patch n Patch n Patch (contacted) throughs SE (contacted) throughs SE (contacted) throughs SE Positive script 518 20.8% 1.8 494 20.2% 1.8 1,025 22.9% 1.3 Negative script 522 19.3% 1.7 480 26.0% 2.0 1,032 23.3% 1.3 Difference (percentage points) -1.5 2.5 5.8* 2.7 0.3 1.9 Note: * p<0.05 12 American Politics Research 00(0) estimate. However, these differences cannot account satisfactorily for the failure to observe a large positive effect with the same design and exact same population, but a different issue and treatment scripts in Field Experiments 3 and 4. Field Experiments 3 and 4 share identical design, treatments, and subject populations and differ from each other in timing, so any post hoc explanations for the divergent findings must be based on timing. As the campaigns were precipitated by proposals introduced by the governor and not the electoral calendar, there is no obvious reason to believe timing can account for the difference. A post hoc examination of newspapers finds very little coverage of the issue at either time so public attention to the issue seems unlikely to explain the difference across the two experiments.7 Our partner organization could not identify any other differences in the political context across the two experiments as possible explanations. We suspect that the "statistically significant" results we observe in Field Experiment 3 were a function of sampling variability (i.e., Field Experiment 3 was merely a lucky draw from a distribution centered around a much smaller mean). Thus, the next section conducts a meta-analysis to help minimize the noise and observe whether a pattern emerges. Meta-Analysis of Field Experiments I to 4 The treatment effect for each of the four field experiments is displayed in Figure 1. In each experiment, the treatment effect is from Tables 2 and 4: the participation rate (voter turnout or "patching through") among individuals contacted with the negatively framed treatment minus the participation rate among individuals contacted with the positively framed treatment. The bottom of the table reports three random effects meta-analyses of the "Pit Rule" experiments (Experiments 3 and 4, identical except for timing), all three patch-through experiments (Experiments 2-4), and all four experiments. All of the meta-analytic estimates are in the expected direction of negative frame treatments, but none are distinguishable from zero despite the size of the field experiments and replications. As noted above, the combined effect of Experiments 3 and 4 is large and in the expected direction, but not remotely close to statistical significance (b = 2.7, SE = 2.7, p = .499). The metaanalysis of the three patch-through experiments is closer to zero with even smaller confidence intervals (b = 1.3, SE = 2.1, p = .587). The random effects meta-analysis of all four studies is again indistinguishable from zero (b = 0.9; SE = 0.8, p = .344). Regardless of statistical significance, the effects observed in our field experiments are quite small and suggest that the differences between negative and positive framing in field settings are not stark (see Supplemental Table S3 for Cohen's d statistics for each estimated effect (Cohen, 1988)). Mann et al. 13 Experiment 1: 2010 Voter Turnout - Experiment 2: "Pollution" Patch Through - Exp. 3: "Pit Rule" Patch Through (Jan 2012) - Exp. 4: "Pit Rule" Patch Through (April 12) - Meta analysis: "Pit Rule" Patch Through (Exp 3 & 4) - Meta analysis: All Patch Through - Meta analysis: All Four Experiments -.05 0 .05 .1 Effects reported in percentage points with 95% confidence intervals. Pooled estimates are based on random effects meta-analysis. Results tables available in the supplemental materials. Figure I. Negative frame treatment effect estimates and 95% confidence intervals for Field Experiments I to 4. Conclusion In the four field experiments, we found little-if-any evidence that negative frames about policy returns to participation are substantially better motivators of political behavior in real-world collective action settings than positive frames about policy returns to participation. These modest differences held for both delayed actions such as voter turnout and immediate actions such as contacting a representative. Our findings contrast with the evidence that negative loss-framed messages increase other regarding behavior in public goods games conducted in laboratory settings (e.g., Brewer & Kramer, 1986) or increase healthy behaviors in real-world settings (e.g., Levin et al., 1998). The patch-through calls (Field Experiments 2-4) eliminate the possibility that temporal decay in effect accounts for the difference between laboratory and field by measuring real-world behavior immediately after delivering the stimulus. Moreover, the references to status quo policy in Field Experiments 2 to 4 made them an "easy case" for finding effects from negatively framed messages about the policy returns to participation, but we still find little evidence supporting the negative frame hypothesis for motivating participation in collective action in politics. 14 American Politics Research 00(0) That said, our findings are consistent with a meta-analysis of campaign advertising studies showing little support for the notion that negative advertising has differential effects on voter turnout (Lau et al., 2007) as well as a previous field experiment that failed to uncover a status quo bias effect with respect to contacting the president of the United States (Miller & Krosnick, 2004). Null findings such as these suggest that the robust findings from laboratory settings are washed out in real-world political settings. Heterogeneous response to treatment across subpopulations and circumstances may exist, but using likely demographic characteristics (e.g., age, gender), we were unable to identify subgroups especially responsive to negative frames. The small overall average treatment effect found in our experiments indicates that either negative frames are efficacious among a small portion of the experimental populations (attenuated by a large portion with no effect) or large effects of the negative frame among one subgroup are off-set by a group more responsive to positive frames.8 Even if this heterogeneity exists, the field experiments are starkly different than laboratory findings. The difference from laboratory experiments is especially striking because three of the field experiments use a novel type of real-world political behavior: patch-through calls to elected officials. These policy advocacy calls are a real-world collective action highly similar to the hypothetical statements about policy support used in many laboratory framing experiments in political science. Beyond simply demonstrating the viability of patch-through call field experiments, we believe the similarity between laboratory designs and patch-through calls suggests considerable value for future patch-through call field experiments. These findings raise intriguing questions about motivations for participating in collective action in politics. The difference between field and laboratory experiments may simply indicate the difficulty of generalizing from the lab to the real world. In particular, the stylized games used in many laboratory experiments may not reflect real-world decision-making processes. Different wording of the frames in field experiments might yield larger results, but we found no effect despite attempting to strengthen the manipulations in later experiments and the fact that independent samples offered evidence that our manipulations "worked." The language in our treatments was constrained by legal and political considerations of our partner organization that we believe are commonplace in political communication. If eliciting larger results requires negative frames exceeding these limits, then negative framing cannot provide a general tool for motivating political action, because such frames are unlikely to be employed in normal political discourse. Moreover, a key point of the negative framing literature is the importance of subtle differences rather than heavy-handed rhetoric. Field experiments like ours cannot prove a null hypothesis and only additional field experiments Mann et al. 15 on this topic can determine if alternative wording of the frames will be more successful. Further investigation may uncover negative and positive frames about policy returns to participation that influence real-world political behavior and those would be exciting findings. However, our experiments suggest that any treatment effects estimated will be modest, so large sample sizes will be required to provide convincing evidence of real-world consequences from using negative and positive frames. Authors's note David W. Nickerson is now affilaited with Temple University, Philadelphia, PA, USA. Acknowledgments The authors thank the anonymous reviewers, Kevin Collins, Donald Green, Peter Loewen, John Love, and the participants in the Rethinking Ways to Increase Voter Turnout Workshop at Princeton University for helpful comments for helpful comments. They are grateful for the cooperation of their partner organizations. The experiments were reviewed by the University of Miami institutional review board, protocols #20110124 and #20111102. Data and additional materials are available on Dataverse at https://doi.org/10.7910/DVN/LVBEJK and authors' websites. Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Kevin Arceneaux and David W. Nickerson received no funding related to this article. Christopher B. Mann received payment from the partner organizations for time spent overseeing the random assignment and assessing the impact of the treatments. Mann's funding agreement guaranteed unrestricted rights to publish using the data from the experiments. Mann received no payment related to preparation or publication of the article. Notes 1. See the supplemental materials for a full description of experimental populations, treatment administration, and complete phone scripts for each experiment. 2. The design also included a control group to which no phone call was attempted (n = 80,046) and a placebo script with no positive or negative frame (n = 43,164). These randomly assigned conditions are not relevant to estimating the effect of the negative frame treatment compared with the positive frame treatment, so we do not include them in the analysis (see Note 5 for further information). 16 American Politics Research 00(0) 3. The partnership agreements with the partner organizations for Field Experiment 1 and Field Experiments 2 to 4 specified unrestricted publication rights using the data from these experiments, thus avoiding the potential for selection bias in reported results when organizations control the release of information (Nickerson, 2011; Nickerson & Hyde, 2015). 4. The turnout for all records assigned to the control group was 36.3%. Compared to the control group, the negative frame treatment increased turnout by 0.6 percentage points (SE = 0.3,_ponetailed = .050) among all assigned records. Compared to the control group, the positive frame treatment appeared to increase turnout by 0.3 percentage points (SE = 0.3, ^ne tailed = -202) among all assigned records. The placebo treatment (no positive or negative frame) had no effect on turnout compared with the control group (b = 0.04 percentage points, SE = 0.3, p = .442) among all assigned records. The larger intent-to-treat effects with either framing suggest the framing element of the treatment scripts was noticeable and strong enough to alter behavior. 5. We are not suggesting attenuation of the effect of voter mobilization contact as both treatments increased turnout relative to the control group, as expected from prior research (Bhatti, Dahlgaard, Hansen, & Hansen, 2017; Murray & Matland, 2014; Nickerson, 2007; Panagopoulos, 2011b). 6. Field Experiment 2 utilized the first 13,439 records in the experimental population. Field Experiment 3 utilized the next 13,781 records and Field Experiment 4 utilized the final 33,399 records. 7. The paper of record in the state had one 300 word Associated press story in the Metro section in January 2012 (Experiment 3) and no coverage in April 2012 (Experiment 4). 8. It is also possible that the experimental populations selected by our partner organizations included people who were not sensitive to negative information. At a minimum, the null results indicate that a broad segment of the population, which our partner organizations deemed their most important targets for communication, is not motivated by negative framing to participate in politics. Further studies are needed to confirm broader generalizability. Supplemental Material Supplemental material for this article is available online. ORCID iD Christopher Mann © https://orcid.org/0000-0003-3625-4152 References Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179-211. Arceneaux, K. (2012). Cognitive biases and the strength of political arguments. American Journal of Political Science, 56, 271-285. Mann et al. 17 Arceneaux, K., & Nickerson, D. (2010). Comparing negative and positive campaign messages: Evidence from two field experiments. American Politics Research, 38, 54-83. Bhatti, Y., Dahlgaard, J. O., Hansen, J. H., & Hansen, K. M. (2017). Moving the campaign from the front door to the front pocket: Field experimental evidence on the effect of phrasing and timing of text messages on voter turnout. Journal of Elections, Public Opinion and Parties, 27, 291-310. Bhatti, Y., Dahlgaard, J. O., Hansen, J. H., & Hansen, K. M. (2018). Can governments use get out the vote letters to solve Europe's turnout crisis? Evidence from a field experiment. West European Politics, 41, 240-260. Brewer, M. B., & Kramer, R. M. (1986). Choice behavior in social dilemmas: Effects of social identity, group size, and decision framing. Journal of Personality and Social Psychology, 50, 543-549. Chong, D., & Druckman, J. N. (2010). Dynamic public opinion: Communication effects over time. American Political Science Review, 104, 663-680. Cobb, M. D., & Kuklinski, J. H. (1997). Changing minds: Political arguments and political persuasion. American Journal of Political Science, 41, 88-121. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum. De Dreu, C. K. W., & McCusker, C. (1997). Gain-loss frames and cooperation in two-person social dilemmas: A transformational analysis. Journal of Personality and Social Psychology, 72, 1093-1106. Druckman, J. N. (2004). Political preference formation: Competition, deliberation, and the (ir) relevance of framing effects. American Political Science Review, 98, 671-681. Gerber, A. S., Green, D. P., & Larimer, C. W. (2008). Social pressure and voter turnout: Evidence from a large-scale field experiment. American Political Science Review, 102, 33-48. Gerber, A. S., Huber, G. A., Fang, A. H., & Reardon, C. E. (2018). The comparative effectiveness on turnout of positively versus negatively framed descriptive norms in mobilization campaigns. American Politics Research, 46, 996-1011. Green, D. P., & Gerber, A. S. (2015). Get out the vote! (3rd ed.). Washington, DC: Brookings. Ha, S. E., & Karlan, D. S. (2009). Get-out-the-vote phone calls: Does quality matter? American Politics Research, 37, 353-369. Hibbing, J. R., Smith, K. B., & Alford, J. R. (2014). Differences in negativity bias underlie variations in political ideology. Behavioral and Brain Sciences, 37, 297-307. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263-292. Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1990). Experimental tests of the endowment effect and the Coase theorem. Journal of Political Economy, 98, 1325-1348. Kalla, J. L., & Broockman, D. E. (2018). The minimal persuasive effects of campaign contact in general elections: Evidence from 49 field experiments. American Political Science Review, 112, 148-166. 18 American Politics Research 00(0) Krishnamurthya, S., Bottomb, W. P., & Rao, A. G. (2003). Adaptive aspirations and contributions to a public good: Generic advertising as a response to decline. Organizational Behavior and Human Decision Processes, 92(1-2), 22-33. Lau, R. R., Sigelman, L., & Rovner, I. B. (2007). The effects of negative political campaigns: A meta-analytic reassessment. Journal of Politics, 69, 1176-1209. Levin, I. P., Schneider, S. L., & Gaeth, G. J. (1998). All frames are not created equal: A typology and critical analysis of framing effects. Organizational Behavior and Human Decision Processes, 76, 149-188. Mann, C. B. (2010). Is there backlash to social pressure? A large-scale field experiment on voter mobilization. Political Behavior, 32, 387-408. Mann, C. B., & Klofstad, C. A. (2015). The role of call quality in voter mobilization: Implications for electoral outcomes and experimental design. Political Behavior, 37, 135-154. McCusker, C, & Carnevale, P. J. (1995). Frame and loss aversion in social dilemmas. Organizational Behavior and Human Decision Processes, 61, 190-201. Meyerowitz, B. E., & Chaiken, S. (1987). The effect of message framing on breast self-examination attitudes, intentions, and behavior. Journal of Personality and Social Psychology, 52, 500-510. Miller, J. M., & Krosnick, J. A. (2004). Threat as a motivator of political activism: A field experiment. Political Psychology, 25, 507-523. Murray, G. R., & Matland, R. E. (2014). Mobilization effects using mail: Social pressure, descriptive norms, and timing. Political Research Quarterly, 67, 304-319. Nicker son, D. W. (2006). Volunteer phone calls can increase turnout. American Politics Research, 34, 271-292. Nickerson, D. W. (2007). Quality is job one: Professional and volunteer voter mobilization calls. American Journal of Political Science, 51, 269-282. Nickerson, D. W. (2011). When the client owns the data. The Experimental Political Scientist: Newsletter of the APS A Experimental Section, 2(2), 5-6. Nickerson, D. W., & Hyde, S. D. (2015). Conducting research with NGOs: Relevant counterfactuals from the perspective of subjects. In S. Desposato (Ed.), Ethics and experiments: Problems and solutions for social scientists and policy professionals (pp. 198-216). New York, NY: Routledge. Nickerson, D. W., & Rogers, T. (2010). Do you have a voting plan? Implementation, intentions, voter turnout, and organic plan making. Psychological Science, 21, 194-199. Panagopoulos, C. (2010). Affect, social pressure and prosocial motivation: Field experimental evidence of the mobilizing effects of pride, shame and publicizing voting behavior. Political Behavior, 32, 369-386. Panagopoulos, C. (2011a). Thank you for voting: Gratitude expression and voter mobilization. The Journal of Politics, 73, 707-717. Panagopoulos, C. (2011b). Timing is everything? Primacy and recency effects in voter mobilization campaigns. Political Behavior, 33, 79-93. Quattrone, G. A., & Tversky, A. (1988). Contrasting rational and psychological analyses of political choice. American Political Science Review, 82, 719-736. Mann et al. 19 Rosenthal, R. (1979). The file drawer problem and the tolerance of null results. Psychological Bulletin, 86, 638-641. Sell, J., & Son, Y. (1997). Comparing public goods and common pool resources: Three experiments. Social Psychology Quarterly, 60, 118-137. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453-458. Valenzuela, A., & Michelson, M. (2016). Turnout, status, and identity: Mobilizing Latinos to vote with group appeals. American Political Science Review, 110, 615-630. Van Dijk, E., & Wilke, H. (2000). Decision-induced focusing in social dilemmas: Give-some, keep-some, take-some, and leave-some dilemmas. Journal of Personality and Social Psychology, 78, 92-104. Author Biographies Christopher B. Mann is an assistant professor of Political Science at Skidmore College in Saratoga Springs, NY. His research focuses on field experiments on increasing political participation and on the the impact of changes in election administration on political participation. Kevin Arceneaux is Thomas J. Freaney, Jr. Professor of Political Science, Faculty Affiliate with the Institute for Public Affairs, and director of the Behavioral Foundations Lab at Temple University. He studies how people make political decisions, paying particular attention to the effects of psychological biases. David W. Nickerson (PhD Yale 2005) is an associate professor in the Department of Political Science at Temple University. His research focuses substantively on how campaigns mobilize supporters and the effectiveness of these tactics.