Field experiment evidence of substantive,
attributional, and behavioral persuasion by
members of Congress in online town halls
William Minozzia,1
, Michael A. Nebloa
, Kevin M. Esterlingb
, and David M. J. Lazerc,d
a
Department of Political Science, The Ohio State University, Columbus, OH 43201; b
Department of Political Science, University of California, Riverside,
CA 92521; c
Department of Political Science and College of Computer and Information Science, Northeastern University, Boston, MA 02115;
and d
John F. Kennedy School of Government, Harvard University, Cambridge, MA 02138
Edited* by H. Russell Bernard, University of Florida, Gainesville, FL, and approved January 29, 2015 (received for review September 25, 2014)
Do leaders persuade? Social scientists have long studied the
relationship between elite behavior and mass opinion. However,
there is surprisingly little evidence regarding direct persuasion by
leaders. Here we show that political leaders can persuade their constituents
directly on three dimensions: substantive attitudes regarding
policy issues, attributions regarding the leaders’ qualities, and
subsequent voting behavior. We ran two randomized controlled field
experiments testing the causal effects of directly interacting with a
sitting politician. Our experiments consist of 20 online town hall
meetings with members of Congress conducted in 2006 and 2008.
Study 1 examined 19 small meetings with members of the House
of Representatives (average 20 participants per town hall). Study
2 examined a large (175 participants) town hall with a senator. In
both experiments we find that participating has significant and substantively
important causal effects on all three dimensions of persuasion
but no such effects on issues that were not discussed extensively
in the sessions. Further, persuasion was not driven solely by changes
in copartisans’ attitudes; the effects were consistent across groups.
persuasion | field experiment | political science | immigration | terrorism
As thinkers ranging from Aristotle (1) to our own day have
argued, persuasion—a change in the attitude or behavior of
an individual caused by an appeal from a political elite—is integral
to leadership. Although the question of persuasion by leaders is
relevant to almost every form of collective human behavior over
time and place, there is remarkably little evidence that unmediated,
interpersonal appeals from specific leaders affect the attitudes or
behavior of individuals. In contrast, the link between indirect,
mediated persuasion and mass opinion has been studied intensively
(2, 3), with one long-standing literature arguing that elites
generally have a minimal impact on mass opinion (4). Others
argue that the causal arrow is actually reversed—that political
elites are particularly adept at calibrating their statements and
political aims in response to their followers’ opinions (5, 6).
One likely reason for this impasse is that virtually all of the
evidence we might use to measure the effects of persuasive
appeals by leaders is either indirect and observational, or based
on laboratory experiments that only simulate a few features of
real elite–mass interactions. Political scientists and psychologists
have accumulated aggregate-level evidence of elite persuasion by
studying mass media messages and advertising (7, 8), large-N
surveys (2, 5, 9–14), and laboratory experiments with hypothetical
elite–mass interactions (15, 16). However, there is little
publicly available evidence that speaks directly to whether and
how individual elites directly persuade their individual constituents.
Although the contours of such persuasion may mirror
those in the aggregate, and hypothetical scenarios offered in
laboratory settings might yield dynamics similar to those in real
political interactions, there is reason to think that they do not
reliably track each other (17, 18). For example, Neblo et al. (19)
demonstrate a sharp reversal in attitudes and behavior regarding
government and political participation.
Interestingly, as well documented in The Victory Lab (20), there
has been a proliferation of proprietary field experimental research
in election campaigns testing candidates’ ability to persuade and
mobilize voters—for example, Rick Perry, in his reelection campaign
in 2010, was randomly assigned to certain campaign stops to
evaluate impact on voter turnout (20). In contrast, we are aware of
no publicly available evidence that appeals from specific political
elites affect the attitudes or behaviors of specific citizens, despite
prominent accounts of representative democracy hinging on persuasion
in such relationships. The absence of such evidence is all
the more striking in that elites frequently do appeal directly to their
constituents. Town halls, stump speeches, and personal contact
between individual elites and members of the public are standard
features of modern politics (21, 22).
We identify three dimensions of persuasion: substantive, attributional,
and behavioral. Substantive persuasion involves changes
in attitudes about an issue. Attributional persuasion involves
changes in attitudes about the leader. Behavioral persuasion
involves changes in political participation.
Substantive persuasion has been the focus in the existing literature
(23) and is important because it affects support for
particular decisions and may lead to future behavioral changes.
However, most elected officials’ primary communication goals focus
on presentation of self (21, 22), that is, persuading constituents
regarding their personal qualities (e.g., being trustworthy or competent).
This emphasis on attributional persuasion is necessary for
effective leadership, because positive attributions mean that
Significance
Persuasion is at the core of leadership, especially in a democracy,
yet there is remarkably little evidence that direct appeals from
leaders causally affect the attitudes and behaviors of their followers.
The available evidence is either observational and indirect
or based on laboratory experiments that simulate selected
features of interactions with leaders. We fill this void with two
randomized controlled field experiments in which 12 US representatives
and a senator each met with samples of their constituents
in online town halls. We find evidence of substantial
persuasion effects on specific policy issues, attributions of trust
and approval, and ultimately the decision to vote for the leader.
Author contributions: M.A.N., K.M.E., and D.M.J.L. designed research; M.A.N., K.M.E., and
D.M.J.L. performed research; W.M., M.A.N., K.M.E., and D.M.J.L. analyzed data; and
W.M., M.A.N., K.M.E., and D.M.J.L. wrote the paper.
The authors declare no conflict of interest.
*This Direct Submission article had a prearranged editor.
Freely available online through the PNAS open access option.
Data deposition: Replication materials and survey datasets are available at thedata.
harvard.edu/dvn/dv/CTC.
1
To whom correspondence should be addressed. Email: minozzi.1@osu.edu.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.
1073/pnas.1418188112/-/DCSupplemental.
www.pnas.org/cgi/doi/10.1073/pnas.1418188112 PNAS | March 31, 2015 | vol. 112 | no. 13 | 3937–3942
POLITICAL
SCIENCES
ambiguous events will be interpreted to the leader’s benefit,
facilitating survival and providing some freedom for movement
politically. Behavioral persuasion is necessary for leaders to
stay in power—to mobilize voting in reelection. There is an
enormous literature on political behavior (e.g., the correlates of
voting behavior and participation more generally) (24), the role
that elections play in mobilizing or demobilizing voters, and the
role that networks play in mobilizing other forms of political
action (25, 26). However, relatively little has been written on
the behavioral effects of direct appeals from elites.
Oftentimes, on topics such as immigration and terrorism, democratic
politics centers on questions of good and bad, or right and
wrong, and uses of governmental power and hence cannot be solved
by coordination among constituents who share common interests
(27). In these situations, leaders must use direct appeals to persuade
their constituents. Following Aristotle, we identify three mechanisms
that leaders rely on: providing good reasons (logos), invocation
of authority (ethos), and activating emotions (pathos).
These kinds of activities affect listeners on a dyadic basis, with
the action of the leader directly altering the attitude or behavior
of the follower via one or more mechanisms.
To study direct interpersonal persuasion by political elites we
designed two field experiments, in which members of the public
interacted directly with their members of Congress (MOCs) in an
online town hall meeting. Here we follow the path of recent field
experimental studies of peer effects (28, 29), which use randomization
to identify treatment effects in naturalistic online
settings. Specifically, we recruited sitting members of the US
House of Representatives (study 1) and a US senator (study 2) to
interact with their constituents via a real-time online forum (voice
mediated with real-time transcription). Participants could address
their MOC and listen to their member’s responses to the questions
and comments posed by the group. The participants were recruited
from high-quality district samples (Supporting Information) and were
compensated. Constituents were randomly assigned to receive
reading material and participate in the discussion or to a control
group that only received the reading material. Using responses to
pretreatment and posttreatment survey questions, we test whether
MOCs were effective in their persuasive appeals–that is, the hypothesis
that, on average, meeting with a political elite changes
attitudes and behaviors of members of the public in the direction
sought by the elite.
Results
Experimental Design. Studies 1 and 2 used a similar research design,
in which experimental subjects who expressed willingness to
participate in an online forum with their MOC were randomly
assigned to either a treatment or control condition. In the control
condition, here referred to as “information only,” participants
read background materials about the issue (background materials
are included in Supporting Information). In the treatment condition,
referred to as the “deliberative session,” in addition to
reading the background materials participants were invited to attend
an online town hall meeting with their MOC. Information in
the background materials was drawn from nonpartisan sources
(e.g., Congressional Research Service and Office of Management
and Budget reports), edited to a ninth-grade reading level, and
vetted by the participating MOCs’ staffers (the background
materials appear in Supporting Information). Both the information
only and deliberative session treatments succeeded in increasing
knowledge on immigration (17), meaning that subjects actually
read and retained much of the policy information assigned to
them. Subjects who attended the deliberative session were more
likely to do so than those who were assigned to information only
or did not comply with their assignment to attend the session.
Each session focused on a single policy issue. Discussion in
study 1 focused on the issue of immigration and in study 2 on
terrorist and detainee policy. Sessions were lightly moderated by
one of the authors. During each session, constituents typed comments
and questions into an online discussion platform. After
reviewing these contributions, a screener posted them to the whole
group in approximately the order in which they were received. The
screener played no active role in facilitating the discussion and had
no knowledge of the study hypotheses or the content of the surveys.
Questions were screened if they were duplicative of a prior
question. The MOC responded through a telephone linked to
a computer. Constituents received the MOC’s responses by listening
over computer speakers and/or reading a real-time transcription.
After 35 min, the MOC and staff logged off. In study 1,
the constituents were then directed to a chat room to have an
open-ended discussion, which lasted 25 min. In study 2, the main
session was extended and the chat session dropped because the
larger number of participants made a plenary chat impractical.
Fig. 1 illustrates one of the sessions with Representative George
Radanovich (R-CA). Sample video is available in Supporting
Information.
We are interested in substantive, attributional, and behavioral
persuasion, and therefore we asked participants to complete
questionnaires before and after the sessions. Subjects received
background materials 1 wk after the baseline questionnaire, and
sessions were held 1 wk after that. A postsession survey was
fielded 1 wk after the session, and a final survey was fielded
the day after the subsequent November election. In study 1 the
sessions were held over several months, so the time between
the postsession survey and the postelection survey varied from 1 to
4 mo (median = 60 d). In study 2, the session was held in July,
meaning a time lapse of 3 mo between postsession survey and
postelection survey. The items used to measure substantive and
attributional persuasion are drawn from the follow-up surveys,
and the item used to measure behavioral persuasion is from the
postelection survey.
For substantive persuasion, we asked participants policy questions
that related to the topic of the sessions, and we recoded
responses within each MOC–constituent dyad so that higher values
indicate more agreement between the two. To determine MOC
positions, we relied on statements made during the deliberative
sessions and on statements made in other contexts. Each MOC
either strongly agreed or strongly disagreed with a given policy,
so a move in either direction unambiguously constituted more or
less agreement. For attributional persuasion, we asked participants
whether they trusted or approved of their MOC. For behavioral
persuasion, we asked whether participants intended to
vote for their MOC in the upcoming election. Finally, the survey
fielded the day after the election asked participants how they
actually voted.
Fig. 1. A screen capture from the experimental interface. The image depicts
what a subject saw during one of the experimental sessions.
3938 | www.pnas.org/cgi/doi/10.1073/pnas.1418188112 Minozzi et al.
Study 1. Study 1 includes 19 sessions with 12 MOCs that took
place between June and October 2006 (Table 1). There was robust
variation among MOCs: five Republicans and seven Democrats,
spread across all four major geographical regions, two women,
an African American, and representatives of both parties’ leadership.
All were running for reelection, and they were diverse
ideologically, including one MOC from each party who voted
against their party on the topic under discussion (i.e., recent
immigration legislation).
Participants in study 1 were recruited by an online survey firm
from a probability sample panel of survey respondents designed
to be demographically representative of the US population.
Given the novelty of the study, before we ran the study we did
not have sufficient information to reliably estimate compliance
rates. Therefore, we used a two-step process to assign each participant
to a treatment condition. All participants completed
a baseline survey in which they were asked a filter question about
whether they would like to participate in the deliberative session,
to complete surveys only, or to opt out entirely. Here, we focus
exclusively on subjects who were willing to participate. Of the
2,237 participants who completed this initial survey, 1,566 indicated
that they would like to participate. Of these, 1,259 participants
were then randomly assigned to either the deliberative
session or information only.†
In all, there were 19 small (∼20
people each) group meetings with MOCs.
To differentiate substantive persuasion from attitude change
via other mechanisms, we conducted a placebo test in which we
compare responses on two subtopics, one that received a great
deal of attention in the sessions and another that received almost
no attention. For study 1, the high-attention issue was whether to
create a path to citizenship for undocumented immigrants, and the
low-attention issue was whether to change the number of immigrants
allowed to enter the United States legally. The path to citizenship
issue came up frequently in the sessions, whereas changes
in legal immigration levels received almost no discussion—the
words “amnesty” and “citizenship” appear more than 150 times in
the session transcripts, whereas the phrase “legal immigration”
appears only six times. Moreover, in those cases when the latter
phrase appeared it was obvious from context that the topic of
discussion was actually whether illegal immigrants should be able
“earn” citizenship.
To evaluate the effects of attending the deliberative session,
we used instrumental variables regression to estimate the complier
average causal effect, which is the effect of attending the
session on the attitudes and behavior of participants who would
attend only if assigned (i.e., the compliers) (30, 31). We instrument
for attendance at the deliberative session using invitation
to attend, which was randomly assigned. We also have
responses from the baseline survey for these questions, and we
condition on levels of the pretreatment response, except in the
case of actual vote, for which we condition on pretreatment
vote intent.
Analysis of balance between observational units assigned to
treatment and control revealed a significant difference between
the two groups on whether the participant and MOC belonged to
the same party (Materials and Methods). Imbalance on a single
covariate can be expected simply due to chance, but this particular
covariate is also likely to predict our outcome variables.
Therefore, in study 1 we condition on this covariate as well.
Results from Study 1. The analyses reveal strong evidence of
persuasion on substantive policy questions and on attributions
and behavior toward the MOC. The results are presented in
Table 2 and Fig. 2. The first column of Table 2 lists the complier
average causal effect. Positive numbers represent more persuasion
by the MOC, and measures of effect sizes are presented in
the final column using Cohen’s d.
On path to citizenship, participants who attended the session
moved toward their MOC’s position more than they would have
in the information only condition (P = 0.011; all reported tests
are two-tailed). As expected, however, attendees did not move
significantly toward their MOC on the issue of legal immigration
(P = 0.354). A formal placebo test rejects the null hypothesis of
no difference between these two effects (P = 0.079). Beyond attitudes
on issues, attendees also exhibited changes in their attitudes
toward the member. On average, attendees showed markedly increased
trust (P = 0.003) and approval (P = 0.053). Finally, we find
evidence of behavioral persuasion, most clearly in the 13.8% increase
in intent to vote for the member (P < 0.001). Attendance in
the session with the member also caused a 9.8% increase in
likelihood of voting for the member in the November election,
although this result yields weaker statistical significance (P =
0.183). Moreover, effect sizes are substantial, most notably for
the increase that attending the session had on trust.
We conducted many robustness checks (reported in Materials
and Methods with details in Supporting Information), including
estimation of intent-to-treat effects (which ignore compliance/
attendance and focus simply on the effect of random assignment),
Table 1. Participating members of Congress
Member Party State Study
Rep. Earl Blumenauer D OR 1
Rep. Michael Capuano D MA 1
Rep. James Clyburn D SC 1
Rep. Mike Conaway R TX 1
Rep. Anna Eshoo D CA 1
Rep. Jack Kingston R GA 1
Rep. Zoe Lofgren D CA 1
Rep. Don Manzullo R IL 1
Rep. Jim Matheson D UT 1
Rep. David Price D NC 1
Rep. George Radanovich R CA 1
Rep. Dave Weldon R FL 1
Senator Carl Levin D MI 2
D, Democrat; R, Republican.
Table 2. Experiment results for study 1 (House of
Representatives)
Outcome CACE SE P N d
Policy attitudes
Agree on path to citizenship? 0.144 0.058 0.011 565 0.365
Agree on legal immigration? 0.033 0.036 0.354 566 0.108
Attitudes toward MOC
Trust 0.118 0.039 0.003 492 0.523
Approve 0.076 0.040 0.053 526 0.286
Behavior toward MOC
Vote Intent 0.138 0.038 <0.001 565 0.452
Actual Vote 0.098 0.072 0.183 516 0.197
CACE, instrumental variables regression estimate of complier average
causal effects; d, Cohen’s d; n, number of observations; P, two-tailed bootstrapped
P values; SE, bootstrapped SEs with 10,000 resamples. All variables
have been rescaled 0–1.
†
Some participants were also assigned to a third “true control” group that did not receive
the materials, but we focus on the information only condition as the control group here
because a different survey protocol governed whether they received the follow-up surveys,
which complicates causal inferences. We also assigned 201 participants who indicated
that they were not willing to participate in the deliberative session to receive
information only. Those participants are also excluded in the analysis presented here
because of their unwillingness to participate in the session.
Minozzi et al. PNAS | March 31, 2015 | vol. 112 | no. 13 | 3939
POLITICAL
SCIENCES
tobit models (to account for ceiling/floor effects), analyses of attrition
and missingness in follow-up surveys, and treatment effect
heterogeneity by congressional district. Our findings consistently
support the inferences presented in Table 2.
Study 2. Although the findings in study 1 are strong, several
factors might limit their generalizability. First, the immigration
issue may have been particularly suited to elite persuasion.
Second, the small size of the sessions may have increased the
likelihood or extent of persuasion. Third, because of the novelty
of the study and cost considerations, we used an unusual assignment
procedure (the filter question before random assignment).
We therefore conducted a second study in July 2008 using a
similar research design but on a different issue, with a larger
session, and using a streamlined assignment procedure. The session
was again followed by a survey fielded 1 wk after the event, as
well as a postelection survey 3 mo later, immediately after Election
Day. Participants (n = 900) were recruited from a nonprobability
sample and randomly assigned to meet with their
sitting US senator (Carl Levin, D-MI) in a single online forum.
Of these, 462 were assigned to the deliberative session and 175
actually attended the session to discuss issues surrounding terrorism
(e.g., torture, rendition, and the detainees held at Guantánamo Bay,
Cuba). Measures of treatment group balance are presented in
Materials and Methods.
Study 2 shared several qualities of study 1. Some participants
were randomly assigned to receive only background materials on
the issue, and they serve as our control group.‡
Similarly, we
report estimates of the complier average causal effect (i.e., the
average effect of attendance on attitudes and behavior for participants
who would attend the session only if assigned to do so).
To estimate the effects of treatment (i.e., attendance at the deliberative
session) we again use instrumental variables regressions
(30, 31), and we continue to condition on levels of the
pretreatment response from the baseline survey. Here, we lack
baseline responses not only for actual vote behavior, but also for
the substantive questions. For these outcomes we condition,
respectively, on levels of vote intent and on responses to a general
question about the appropriateness of torture. However, in
study 2 no variables presented balance problems, and therefore
we do not add any further conditioning covariates (Materials and
Methods). Finally, we again isolate substantive persuasion with a
placebo test that compares effects on one subtopic that received
a great deal of attention to one that received almost no attention.
Here, the topic of waterboarding came up frequently during the
session, whereas the topic of whether to close the detainment
facility at Guantánamo did not come up at all.
Results from Study 2. The results from study 2 are presented in
Table 3 and Fig. 3. Attendees moved toward Senator Levin’s
position on waterboarding substantially more than they would
have if they were assigned to the control condition (P = 0.052).
In comparison, attendees moved slightly away from the senator’s
position on the placebo issue of whether to close Guantánamo
(P = 0.300). A formal placebo test indicates that these two estimates
were different from each other (P = 0.009). Attendees also
exhibited changes in their attitudes toward the senator, showing
increases of about 11% of the scales for trust (P = 0.002) and
approval (P = 0.001). Finally, we also find evidence of behavioral
persuasion; there was a 10.5% increase in intent to vote for the
senator (P = 0.004) and an even larger 13.1% increase in November
(P = 0.073).
Study 2 also presented an opportunity to measure the decay of
the effects of attending the session. The postelection survey in
study 2 included the approval question, which allows a comparison
of effects 1 wk and 3 mo postsession. The estimated effect
1 wk after the session was 0.107 (Table 3). Using the postelection
measure of approval, the estimated effect of attendance at the
deliberative session was 0.049 (SE = 0.046, P = 0.303, n = 577).
The decline of 0.058, although substantial, does not rise to the
level of statistical significance (P = 0.254). This finding indicates
that the long-term effect of attendance at these sessions is surprisingly
strong, at least on attributional persuasion.
We also conducted robustness checks to estimate intent-totreat
effects and account for attrition and ceiling/floor effects
and reanalyzed voting behavior using validated votes (32). These
results broadly support the inferences presented in Table 3.
Fig. 2. Persuasion by members of the US House of Representatives. Complier
average causal effects and 95% confidence intervals for subjects who
would attend the online town hall if assigned to do so. Confidence intervals
are constructed using the 2.5% and 97.5% quantiles from 10,000 bootstrap
replicates. From top to bottom, n = 565, 492, 526, 565, and 516.
Fig. 3. Persuasion by Senator Levin. Complier average causal effects and
95% confidence intervals for subjects who would attend the online town
hall if assigned to do so. Confidence intervals are constructed using the 2.5%
and 97.5% quantiles from 10,000 bootstrap replicates. From top to bottom,
n = 495, 501, 487, 500, and 574.
‡
As in the first study, participants could also be assigned to a third “true control”
group. In this article, we focus exclusively on subjects who were willing to participate
in a deliberative session and who were assigned either to deliberative session or
information only.
3940 | www.pnas.org/cgi/doi/10.1073/pnas.1418188112 Minozzi et al.
Discussion
To date there is surprisingly little evidence for direct, interpersonal
persuasion by leaders. In this paper we have reported randomized
controlled field experimental evidence that leaders (sitting MOCs
and a senator) were able to persuade their followers (their constituents)
regarding their opinions on policy questions, attributions
about the leader, and their behavior (in this case, voting for the
leader at increased rates) by meeting with and engaging them in
online town halls.
However, another possibility is that these changes in attitudes
and behavior are not reflective of persuasion, but instead demonstrate
sorting. That is, perhaps the only thing that occurred
during the online sessions is that participants who were politically
ignorant but nonetheless receptive to their MOC’s message
learned what they “should” think. For example, constituents who
were members of the MOC’s party might have learned their
party’s position on the issue under discussion, or perhaps they
learned that their MOC was also an adherent of their own political
ideology or party. Perhaps some participants even learned that
there was an upcoming election during which they could vote.
There are at least two reasons that this possibility is unlikely.
First, if sorting were taking place, then participants who were not
copartisans of the leader should also have been able to sort based
on their involvement in the sessions, yet we see positive effects
overall. Second, a series of conditional effects analyses reveals no
systematic pattern (Fig. 4). Here, we included interaction terms
to capture the moderating effect of shared partisanship within
the member–constituent dyad. The majority of the differences in
causal effects are not statistically significant, meaning that there
are few detectable differences between how copartisans and
other constituents reacted to their leaders during and after the
sessions. Moreover, the few significant differences are not consistent
across studies; they occur on different dimensions of
persuasion. These significant effects actually indicate smaller
effects for copartisans than for others. Similarly, there are neither
significant nor consistent conditional effects based on differences
in education, which further bolsters the claim that these
estimates represent genuine persuasion. Further research should
focus not only on the moderators of persuasion but also its mechanisms,
which might include the provision of reasons, appeals to
authority, or emotional stimuli.
Importantly, these results apply to a specific population—
constituents who were given background material, who were
likely to attend a deliberative session, and who were compensated
for their time—and not necessarily to citizens more generally.
However, one positive sign from this study is that active citizens,
the ones who are open to attend town hall meetings, are also open
to new arguments and positions from elites. If deliberative sessions
like these were to become more popular, it is likely that they would
resemble these sessions, perhaps only differing in compensation.
Although our setting is distinctly the 21st century—representatives
talking to a dispersed group of constituents via new
communication technology—persuasion is of central importance
for leadership for any time or place. All leaders must decide
whether it is worth their time to meet directly with their followers,
rather than communicate solely via broadcast (e.g., through mass
media). Our findings provide reason to think that it is worth it.
Direct interaction can yield distinct and substantial persuasion on
several dimensions that matter to leaders, and our forums proved
scalable, ranging from small groups to nearly 200 participants.
Materials and Methods
Treatment Group Balance. We contracted online survey research firms to draw
samples. The survey firms provided responses to questions about background
characteristics, and our baseline survey asked others including several of
our outcomes of interest. We tested for treatment group balance using the
xBalance command from the R package RItools (33). In study 1, an omnibus
test for balance (within congressional districts) indicates that the
Table 3. Experiment results for study 2 (Senator Levin)
Outcome CACE SE P N d
Policy attitudes
Agree on waterboarding? 0.092 0.047 0.052 495 0.232
Agree on closing Guantánamo? −0.054 0.053 0.300 497 −0.146
Attitudes toward Levin
Trust 0.105 0.031 0.002 501 0.386
Approve (March survey) 0.107 0.034 0.001 487 0.330
Behavior toward Levin
Vote Intent 0.105 0.036 0.004 500 0.292
Actual Vote 0.131 0.076 0.073 574 0.262
CACE, instrumental variables regression estimate of complier average
causal effects; d, Cohen’s d; n, number of observations; P, two-tailed bootstrapped
P values; SE, bootstrapped SEs with 10,000 resamples. All variables
have been rescaled 0–1. Supporting Information.
Fig. 4. Conditional effects by copartisanship. Complier average causal
effects and 95% confidence intervals for subjects who identify with leader’s
political party (filled squares) and all others (open circles). Confidence intervals
are constructed using the 2.5% and 97.5% quantiles from 10,000 bootstrap
replicates. (A) From top to bottom, n = 565, 492, 526, 565, and 516. P values
from bootstrapped tests of differences in causal effects between same party
identifiers and all others are P = 0.583, 0.074, 0.470, 0.823, and 0.879. (B) n =
495, 501, 487, 500, and 574. P values from bootstrapped tests of differences in
causal effects are P = 0.061, 0.289, 0.919, 0.156, and 0.068.
Minozzi et al. PNAS | March 31, 2015 | vol. 112 | no. 13 | 3941
POLITICAL
SCIENCES
randomization was successful (χ2
= 18.1 on 22 df, P = 0.697). Slight imbalances
appeared on several individual covariates. In one case, whether the
MOC and the constituent share a political party, imbalance was relatively
large. Imbalance on a single covariate is likely based on chance alone. However,
because this particular covariate is associated with our potential outcomes of
interest, we condition on an indicator for shared partisanship within the MOC–
constituent dyad in our analyses for study 1. In study 2, an omnibus test for
balance indicates successful randomization (χ2
= 18.8 on 18 df, P = 0.403).
Only slight imbalances appear for individual covariates in study 2. Details
on recruitment, questions, and balance are in Supporting Information.
Estimation of Causal Effects. We use instrumental variables regression to
estimate complier average causal effects, or the effect of attending the
session for participants who would attend only if invited to do so. We use
attendance as the endogenous regressor and random assignment as the
instrument. Where possible, we also condition on indicator variables for
baseline response levels (including missingness). The only exception is for
analysis for the questions on waterboarding and Guantánamo in study 2, and
for Actual Vote, for which we had no baseline questions. In those cases we
used baseline responses to a general question about whether torture is ever
justifiable and baseline Vote Intent, respectively. Where baseline responses
were available, we also conducted change score analysis, which confirms the
results in the main text. All instrumental variables regressions were calculated
with the ivreg command from the R package AER (34).
We rely on standard assumptions to warrant causal inferences (30, 31),
most notably excludability of assignment and the stable unit treatment
value assumption (SUTVA). In study 1, excludability is threatened by the
association between assignment and whether a participant identified as
a member of the same party as her MOC. Therefore, in study 1, we condition
on this covariate. We also rely on SUTVA, which is threatened if leaders
acted merely as coordinating devices. In that case, one subject’s potential
outcomes would depend on whether others (with varying ideologies perhaps)
were assigned to attend the session. The changes we document would
then not be persuasion per se, but rather would have occurred if the MOC
had been replaced by any figure. There are several reasons to think that
SUTVA is not violated here. First, the deliberation elicited brief questions
from the experimental subjects rather than opportunities for persuasive
speeches from them, which suggests that it would be difficult for one subject
to move sentiment. Second, survey responses were collected 1 wk after
the session, meaning that any desire to coordinate would need to extend
past the end of the session. Given that it is unlikely that subjects knew each
other personally (or could even identify each other if they did), it is unlikely
that a drive for coordination then motivated responses. Finally, there was
a surprising amount of intersection between sessions in the topics that were
raised. This suggests that the treatment (at least within MOC) was fairly homogeneous
and likely robust to the inclusion or exclusion of particular subjects.
Compliance and Attrition. Treatment noncompliance and attrition in followup
surveys occurred in both studies. For detailed analysis of compliance (i.e.,
predictors of who attended the deliberative sessions) see ref. 21. To look for
evidence that either compliance or attrition was associated with baseline
responses, we stratified observations into four complier-reporter types (attended
session/answered follow-up, attended/did not answer, etc.) and performed
analyses of variance in baseline responses to test for differences across types. In
only 1 case out of the 12 did this association reach conventional significance
levels (noncompliers who responded to follow-up had lower baseline responses
to the trust question in study 2). We also calculated intent-to-treat effects of
random assignment (regardless of attendance). Tests of null hypotheses conducted
on this basis yielded inferences nearly identical to those in Tables 2 and 3.
Attrition varied by question but was more common in study 1. To address attrition,
we first reran analyses reweighting by the inverse probabilities of
responding to the follow-up, which again yielded inferences very similar to those
in Tables 2 and 3. We also conducted a trimming analysis to bound intent-totreat
estimates. The resulting bounds straddle zero in only one case (Path to
Citizenship in study 1). Details appear in Supporting Information.
Heterogeneity of Causal Effects Across Districts in Study 1. Study 1 includes 12
congressional districts, and it is possible that some MOCs were more persuasive
than others. We conducted two analyses to test for this possibility. First, we used
a leave-one-district-out procedure in which we sequentially held out a district
and calculated estimates based on the remaining districts. The SEs from this
procedure are very similar to SEs in Table 2, and the root mean square prediction
errors for each held-out district yield very small differences across
districts. Finally, we estimated a version of the instrumental variables model
with random coefficients at the district level. The random coefficient estimates
exhibited considerable variation by district on some outcomes, but no
district stood out as consistently having larger or smaller estimates across
outcomes. Details appear in Supporting Information. Research protocols were
reviewed and approved by the institutional review boards at The Ohio State
University, Harvard University, and the University of California, Riverside.
ACKNOWLEDGMENTS. The Congressional Management Foundation helped
implement these studies, and we particularly thank Beverly Bell, Nicole Folk
Cooper, Brad Fitch, Kathy Goldschmidt, and Rick Shapiro for their help. We also
thank Curt Ziniel for research assistance and Martin Johnson, Josh Robison, and
the reviewers for their helpful comments. Finally, we thank Representatives
Earl Blumenauer, Michael Capuano, James Clyburn, Mike Conaway, Anna Eshoo,
Jack Kingston, Zoe Lofgren, Don Manzullo, Jim Matheson, David Price, George
Radanovich, and Dave Weldon and Senator Carl Levin and their staffs for their
participation in these studies. We also acknowledge the support of the Ash
Center for Democratic Governance at the Harvard Kennedy School for support
of the data collection for the Levin session. Funding was provided by Grant
IIS-0429452 from the National Science Foundation.
1. Roberts WR (1984) Rhetorica. In The Works of Aristotle, ed Barnes J (Princeton Univ
Press, Princeton, NJ), pp 2152–2269.
2. Page BI, Shapiro RY (1992) The Rational Public (Univ Chicago Press, Chicago).
3. Lupia A, McCubbins MD (1998) The Democratic Dilemma (Cambridge Univ Press, New York).
4. Finkel SF (1993) Reexamining the ‘minimal effects’ model in recent presidential
campaigns. J Polit 55(1):1–21.
5. Erikson RS, MacKuen MB, Stimson JA (2002) The Macro Polity (Cambridge Univ Press,
Cambridge, UK).
6. Geer J (1996) From Tea Leaves to Opinion Polls (Columbia Univ Press, New York).
7. Iyengar S, Kinder DR (1989) News That Matters (Univ Chicago Press, Chicago).
8. Gerber AS, Gimpel JG, Green DP, Shaw DR (2011) How large and long-lasting are the
persuasive effects of televised campaign ads? Results from a randomized field experiment.
Am Polit Sci Rev 105(1):135–150.
9. Converse PE (1964) The nature of belief systems in mass publics. Ideology and Discontent,
ed Apter DE (Free Press, New York), pp 206–261.
10. Zaller J (1992) The Nature and Origin of Mass Opinion (Cambridge Univ Press, New York).
11. Key VOJ (1966) The Responsible Electorate: Rationality in Presidential Voting, 1936–
1960 (Harvard Univ Press, Cambridge, MA).
12. Popkin S (1991) The Reasoning Voter: Communication and Persuasion in Presidential
Campaigns (Univ Chicago Press, Chicago).
13. Adorno TW, Frenkel-Brunswik E, Levison DJ, Sanford RN (1950) The Authoritarian
Personality (Norton, New York).
14. Alvarez RM, Brehm J (2002) Hard Choices, Easy Answers (Princeton Univ Press,
Princeton, NJ).
15. Lodge M, McGraw KM, Stroh P (1989) An impression-driven model of candidate
evaluation. Am Polit Sci Rev 87(2):399–419.
16. Chong DC, Druckman JN (2007) Framing public opinion in competitive democracies.
Am Polit Sci Rev 101(4):637–655.
17. Esterling KE, Neblo MA, Lazer DMJ (2011) Means, motive, and opportunity in becoming
informed about politics: A deliberative field experiment with members of Congress
and their constituents. Pub Op Q 75(3):483–503.
18. Esterling KE, Neblo MA, Lazer DMJ (2011) Estimating treatment effects in the presence
of noncompliance and nonresponse: The generalized endogenous treatment
model. Polit Anal 19(2):205–226.
19. Neblo MA, Esterling KM, Kennedy RP, Lazer DMJ, Sokhey AE (2010) Who wants to
deliberate – and why? Am Polit Sci Rev 104(3):566–583.
20. Issenberg S (2012) The Victory Lab: The Secret Science of Winning Campaigns (Crown,
New York).
21. Fenno RF (1978) Home Style: House Members in Their Districts (Little, Brown, Boston).
22. Bianco WT (1994) Trust: Representatives and Constituents (Univ Michigan Press, Ann
Arbor, MI).
23. Petty RE, Wegener DT, Fabrigar LR (1997) Attitudes and attitude change. Annu Rev
Psychol 48(1):609–647.
24. Burns N, Schlozman KL, Verba S (2001) The Private Roots of Public Action: Gender,
Equality, and Political Participation (Harvard Univ Press, Cambridge, MA).
25. Nickerson DW (2008) Is voting contagious? Evidence from two field experiments. Am
Polit Sci Rev 102(01):49–57.
26. Mutz DC (2004) Cross-cutting social networks: Testing democratic theory in practice.
Am Polit Sci Rev 96(1):111–126.
27. Ahlquist JS, Levi M (2011) Leadership: What it means, what it does, and what we want
to know about it. Annu Rev Polit Sci 14:1–24.
28. Bond RM, et al. (2012) A 61-million-person experiment in social influence and political
mobilization. Nature 489(7415):295–298.
29. Aral S, Walker D (2012) Identifying influential and susceptible members of social
networks. Science 337(6092):337–341.
30. Angrist JD, Imbens GW, Rubin DB (1996) Identification of causal effects using instrumental
variables. J Am Stat Assoc 91(434):444–455.
31. Gerber AS, Green DP (2012) Field Experiments (Norton, New York).
32. Cassel C (2004) Voting records and validated voting studies. Public Opin Q 68(1):102–108.
33. Hansen B, Bowers J (2008) Covariate balance in simple, stratified and clustered
comparative studies. Stat Sci 23(2):219–236.
34. Kleiber C, Zeileis A (2008) Applied Econometrics with R (Springer, New York).
3942 | www.pnas.org/cgi/doi/10.1073/pnas.1418188112 Minozzi et al.