Chapter Three The Identity of the Judge Chapter Two described a fundamental problem with law: it runs out. Conventional legal sources cannot provide a single, objectively correct, answer to every legal question. Some cases are easy, but some are hard; in hard cases, reasonable minds will disagree on the best answer. Attitudinalists insist that in hard cases (at least on the Supreme Court, where most cases are hard, and there is no review of the justices' decisions) it is only judicial ideology that decides cases. But surely that is too simple. As Justice Cardozo and many other judges have acknowledged, a variety of judges' personal characteristics are bound to influence their decisions in hard cases. In this Chapter we examine the empirical evidence on the links between judicial decisions, ideology, and other personal characteristics, including race and gender. Suppose you believe a judge's personal characteristics influenced that judge's decisions. How would you go about trying to prove it? If we are going to examine empirical evidence - and in this book we will look at a lot of it - then we must learn something about empirical methods. Although the use of these methods has proven both challenging and controversial in this one particular area, the methods themselves have had enormous payoff throughout the study of judicial behavior. In addition to discussing whether who is under the robes affects outcomes and the law, this Chapter provides a basic introduction to quantitative methods. I. Introduction: Why Empirical Study A. Professor Revesz and Judge Edwards In the 1990s, Professor (later Dean) Richard L. Revesz—an environmental law expert at New York University School of Law—set out to see whether ideology explains decisionmaking in environmental cases in the United States Court of Appeals for the District of Columbia Circuit (hereafter "D.C. Circuit"). The D.C. Circuit is, arguably, one of the most important courts in the nation. As the administrative state grew in the 1960s and 1970s, a significant fraction of administrative law cases—cases that involve federal agencies that regulate things like airlines, communication, labor, and trade—came to be decided by the D.C. Circuit.1 In some instances, the D.C. Circuit enjoys exclusive jurisdiction, meaning that it is the only court (short of the U.S. Supreme Court) that can 1 John G. Roberts, Jr., What Makes the D.C. Circuit Different? A Historical View, 92 va. Law. Rev 375 (2006). hear appeals from agency action. To get a sense of the stature of the D.C. Circuit, consider that four of the nine Justices sitting on the U.S. Supreme Court today previously served on the D.C. Circuit (Justices Roberts, Scalia, Thomas, and Ginsburg). In addition, Justice Kagan was nominated by President Clinton—but not confirmed—to the D.C. Circuit. Professor Revesz chose to look at cases in which the D.C. Circuit reviewed the work of Environmental Protection Agency (EPA).2 The EPA is the federal agency charged with administering a variety of laws relevant to air, water, and the general environment. In all of the cases Revesz studied, a party was challenging a decision of the EPA: either an industry group was arguing that the EPA had gone too far, or an environmental group was arguing that the agency had not gone far enough. In each case, the court had to decide whether the EPA or the challenger should win. In his study, Professor Revesz compared the percentage of cases where the decision of the EPA was overturned by Democratic and Republican appointees, on both industry and environmental challenges. He divided the cases into periods in which the court's membership was stable, and then chose ten periods to study. Four of the periods (numbered 1 through 4) took place primarily in the 1970s. The remaining six periods (numbered 24, 25, 29, 30, 33, and 34) took place between the mid-1980s and the early 1990s. We present Professor Revesz's results in Figure 1. 2 Richard L. Revesz, Environmental Regulation, Ideology, and the D.C. Circuit, 83 va. L. Rev. 1717 (1997). 2 Democratic Republican Period 33 60 50 40 30 2 Pi o > Period 24 Period 1 60 50 40 30 n Period 34 Period 25 Period 2 Period 29 Period 3 Period 30 Ü 60 50 40 30 Period 4 Wusüy Eirvronmertal Wusüy Eirvronmertal Wusüy Eirvronmertal Wusüy Eirvronmertal Challenge T ype Figure 1. Percentage of votes to reverse the EPA in industry and environmental challenges, by party. Data obtained from Richard Revesz, Environmental Regulation, Ideology, and the D.C. Circuit, 83 VA. L. Rev. 1739, 1742 (1997). Including en banc votes, differences between Democrat and Republican appointees in industry challenges in Periods 25 and 30 are statistically significant at a 95% confidence level; for 3 environmental challenges, differences are statistically significant at a 95% confidence level in Periods 24, 25, 29, 30, 33, and 34. The first thing made clear in the figure is the great deal of variability in the percentage of times that the EPA is reversed. In Period 29, which ran from October 1986 to August 1993, Republican appointees voted to reverse 44% of the time when deciding a challenge brought by an industry group. Democratic appointees, on the other hand, voted to reverse only 26% of the time. In the same time period, Democratic appointees voted to overturn the EPA with environmental challengers 61% of the time; Republican appointees just 32% of the time. In every time period, Republican appointees were more likely to reverse than their Democratic counterparts when industry presented the challenge (the red bars are higher than the blue bars in the left part of each panel). And, in every time period, Democratic appointees were more likely to reverse than their Republican colleagues when an environmental group challenged the EPA. Professor Revesz used statistical analysis to determine whether the differences he observed in the data are "significant." Statistical analysis allows us to determine whether chance alone produced the differences in behavior that we observe, or whether there are differences that cannot be explained by simple randomness. If chance alone might explain what we see, we would conclude that the difference is insignificant. On the other hand, if the difference is really large, we could conclude that the difference is statistically significant—meaning that chance alone cannot account for the findings. You can think of statistical significance as a way to gauge reliability. If a relationship is significant in a sample of cases, you can be pretty sure that it is there. On the other hand, if a relationship is insignificant, you can't be sure that your conclusions would hold up if you looked at a different set of cases. In some of the time periods in Figure 1 the results are insignificant; i.e., there are no reliable differences in behavior between Democratic and Republican appointees. On the other hand, in two time periods there were statistically significant differences between Democratic and Republican appointees in industry challenges, and in six periods there were statistically significant differences in environmental challenges. Based on this evidence, Revesz concluded that "ideology significantly influences judicial decisionmaking on the D.C. Circuit."3 Professor Revesz's study was published in the Virginia Law Review, one of our nation's leading legal publications. The conclusions of Revesz's study also were reported 3 Revesz, Environmental Regulation, Ideology, and the D.C. Circuit, supra, at 1719. 4 in the Corporate Legal Times, a weekly newspaper widely read in legal circles, and in the Weekly StandardA This pioneering study has been cited over four hundred times to date. Yet Professor Revesz's study did not go uncriticized. At some point in the fall of 1998, Judge Edwards published a critique of Professor Revesz's study and another study of the D.C. Circuit performed by Professors Frank Cross and Emerson Tiller.5 (We first met Judge Edwards in Chapter Two, talking about hard and very hard cases.) At the time, Judge Edwards was the Chief Judge of the Circuit. Before becoming a judge, Edwards was a law professor at the University of Michigan and at Harvard. And, as it happens, Judge Edwards is also an Adjunct Professor of Law at New York University School of Law—he is on the same faculty as Revesz! Judge Edwards' critique was scathing. He described the purpose of his essay as follows: This essay . . . aims to debunk the myth that ideology is a principal determinant in decision making on the United States Court of Appeals for the D.C. Circuit. My purpose in writing is to refute the heedless observations of academic scholars who misconstrue and misunderstand the work of the judges of the D.C. Circuit. I will show that, even when one looks carefully at the so-called 'empirical studies' that purport to analyze the work of my Circuit, it is clear that, in most cases, judicial decision making is a principled enterprise that is greatly facilitated by collegiality among judges.6 You can see why Judge Edwards might care. If Professor Revesz's study were to show that judges are always willful policymakers, that would call into question the legitimacy of the court—in this case, the D.C. Circuit on which Judge Edwards serves— and the rule of law. Judge Edwards found the Revesz study to be particularly dangerous in that regard: The Revesz article . . . [has] the potential to engender serious confusion over judicial decision making. Even worse, [this article] may mislead the unsuspecting (who rely on secondhand reports of an article's contents) into thinking that judges are lawless in their decision making, influenced more by personal ideology than legal principles. Where the authors might have one believe that judging is 4 Debbie Mack, Supreme Court Will Hear Nondelegation Issue: A Near-Dormant Doctrine Springs to Life, Corporate Legal Times, Oct. 2000; Bengamin Wittes, Judges and Politics: Cass Sunstein Gets It Wrong, Weekly Standard, Oct. 6,2003. 5 Harry T. Edwards, Collegiality and Decision Making on theD. C. Circuit, supra. 6 Id at 1335. 5 entirely political, I maintain, and always have maintained, that appellate judging is fundamentally a principled practice.7 As is the case with any empirical study, Professor Revesz made a number of choices along the way that might affect what we can learn from his study. That's the nature of empirical studies like Revesz's; there are always certain obstacles. In his criticism, Judge Edwards highlights many of these limitations. In just a moment, we will use Revesz's study and Judge Edwards' critique as a primer on how to read and think about empirical studies of judging, highlighting important things that can and do go wrong in such studies. It will also serve as our introduction to determining how the identity of the judge might affect judicial outcomes and the law. B. Why Quantitative Empiricism Before we do, though, we need to take up a prior question, a fundamental one: why engage in the empirical study of judicial decisionmaking at all? Judge Edwards not only attacked the specific choices Revesz made in his study; he expressed some skepticism regarding the methodology itself: I have no doubt that careful statistical analysis, cautiously interpreted, may conceivably shed some light on judicial decision making. But serious scholars seeking to analyze the work of the courts cannot simply ignore the internal experiences of judges as irrelevant or disingenuously expressed. The qualitative impressions of those engaged in judging must be thoughtfully considered as part of the equation. This is especially true where the data are subject to multiple interpretations. A researcher who assumes the existence of ideological bias or strategic behavior may 'find' that these exist, while a researcher who considers alternative explanations may find that what exists is rather different.8 Judge Edwards' critique reflects a longstanding concern. In the 1960s there was a famous exchange between Wallace Mendelson, who taught public law, and Harold Spaeth, the parent of the attitudinal model, over what was then called Jurimetrics. Mendelson's complaint about the empirical study of judging focused on "behaviorialists' concentration upon what judges do as distinct from what they say. Yet in not knowing (or 7 Id at 1337. 8 Harry T. Edwards, Collegiality and Decision Making on the D. C. Circuit, 84 va. L. Rev. 1335,1338 (1998). 6 not caring) what they say, one is in a poor position to insist upon a discrepancy between their words and deeds."9 To this Spaeth responded as follows: I find the key to judicial behavior in what the justices do, Professor Mendelson in what they say. I focus upon their votes, he upon their opinions. Given the capacity of the human mind for rationalizing behavior, I believe that meaningful analysis is more likely if one focuses upon action, upon decision making, rather than upon what the actors say about what they have done.10 Similarly, Judge Edwards criticizes quantitative empiricists for ignoring the "internal experiences of judges." But, like Spaeth, many researchers worry that what judges say about their work may not be the best evidence of what is actually going on. Indeed, Justice Cardozo, expressed just this worry about his own account of judging, making a pitch for just the sort of empirical study we will be examining here: We may try to see things as objectively as we please. None the less, we can never see them with any eyes except our own. ... I have little hope that I shall be able to state the formula which will rationalize this process for myself much less for others. We must apply to the study of judge-made law [the] method of quantitative analysis .... A richer scholarship than mine is requisite to do the work aright. But until that scholarship is found and enlists itself to the task, there may be a passing interest in an attempt to uncover the nature of the process by oen who is himself an active agent, day by day, in keeping the process alive. That must be my apology for these introspective searchings of the spirit. Richard Posner is a judge on the United States Court of Appeals (in Chicago), an admirer of Cardozo, and a fan of empirical methods. He explains why psychologically, a judge may be loath to accept the results of empirical study of ideology and judging: For judges to acknowledge even just to themselves the political dimensions of their role would open a psychologically unsettling gap between their official job description and their actual job. 9 Wallace Mendelson, An Open Letter to Professor Spaeth and his Jurimetrical Colleagues, 28 J. pol. 429, 429 (1966). 10 Harold J. Spaeth, Jurimetrics and Professor Mendelson: A Troubled Relationship, 27 J. pol. 875 879 (1965). Acknowledging that they were making political choices would also undermine their confidence in the soundness of their decisions, since judges' political choices cannot be justified by reference to their professional background or training. Judges do not like to think that they are expressing an amateurish personal view when they decide a difficult case. Some judges "agonize" over their decisions; most do not; but both sorts feel a psychological compulsion to think they are making the right decision.1 Instead of embracing any single methodology, or rejecting any methodology out of hand, we really need to ask what a particular methodology can accomplish, and what its shortcomings might be. That was the point made by Professor Joseph Tannenhaus, commenting on the Mendelson-Spaeth debate. He pointed out first that everyone's goal was to try to reach some general conclusions from specific data points: In the current controversy over the suitability of quantitative methods for the study of appellate-court behavior, there is a tendency to overlook a rather important similarity among the majority of contenders on both sides. Most contemporary analysts of appellate-court decisions, whether they be lower-court judges, practicing lawyers, journalists, professors of law, or political scientists, tend to comb discrete decisions in a search for uniformities and inconsistencies. However much their motives may vary, analysts of both schools strive to generalize about phenomena which are, in some ways, unique. Utilizing the techniques it considers most apposite, each group collects and classifies data which it hopes to cast into formularies characterizing the behavior of a court and its individual members. Still, Tannenhaus noted a pretty big difference between his "qualifiers" and his "quantififiers:" Fundamental though their common objective may be, the differences between the generalizers who quantify (the quantifiers) and those who do not (the qualifiers) can hardly be put aside. Two of those differences seem presently relevant. In the first place, the quantifier tends to place greater emphasis on systematic and objective classification. He seeks to devise procedures which will permit trained analysts to come up with highly comparable results. On the other hand, the qualifier tends to feel that such striving for reliability sacrifices too much that is vital. In his view the richest ore is mined by those who Richard A. Posner, How Judges Think, supra, at 289. 8 devote their energies to nuances too elusive for systematic objectivity.12 [T]he quantifier is more disposed than the qualifier to study the voting behavior of judges as distinguished from the opinions they father. To the qualifier, a judge's vote grossly oversimplifies the hard choice he is frequently obliged to make among competing principles, values and interests. And what is more, each of a judge's votes is counted equally by the quantifiers, although some decisions are obviously more important than others. How, the qualifier asks, can one equate Korematsu v. United States (sustaining the wartime Japanese evacuation) and Martin v. Struthers (invalidating a city ordinance against doorbell-ringing by peddlers of literature)? Though each case may have involved a fundamental freedom, Korematsu dealt with the physical internment of many thousands of persons, while the Struthers case involved only a minor inconvenience to a small group of proselytizers. A vote against the national government in the evacuation case was of such vastly greater moment than a vote against the city in the doorbell-ringing case that they cannot seriously be treated as equal. Despite these troublesome objections, the quantifier persists in his use of voting data—in part because of the relative ease in recording them in a systematic and ostensibly value-free way. But only in part. Other reasons are, I think, more important.13 Tanenhaus went on to put his finger on what we believe is the core of the matter: Such broadside attacks on quantitative investigation as "thinkers don't count, and counters don't think," or "figures don't lie, but liars do figure," or it is improper to deal with judges as if they were bookmakers, baseball players, gamesmen, or voters in a presidential election, do not face up to the questions which are really fundamental. More appropriate criteria in evaluating a quantitative study of judicial behavior, it would seem to me, are these: (1) Is the study technically sound? (2) Are the findings of real substance? (3) Can the findings be replicated?14 Stated differently, it's hard to write-off an entire approach to inquiry without looking carefully at the findings, and assessing whether they seem to be reliable. Rather, 12 Joseph Tanenhaus, Supreme Court Attitudes Toward Federal Administrative Agencies, 22 J. POL. 502-503 (1960). (emphasis added) 13 Id. at 503. 14 Joseph Tanenhaus, Supreme Court Attitudes Toward Federal Administrative Agencies, supra, at 504. 9 the proof would seem to be in the pudding. The remainder of this book contains a great deal of that pudding. But, as you read, it is critical to ask yourself constantly the three questions Tanenhaus identified. Professors Lee Epstein and Gary King describe the goals of quantitative empirical research in the following way: Regardless of the type of data employed, all empirical research seeks to accomplish one of three ends, or more typically some combination thereof: amassing data for use by the researcher or others; summarizing data so they are easier to comprehend; and making descriptive or causal inferences, which entails using data we have observed to learn about data we would like to gather.15 The advantage of this approach (it has its disadvantages too) is the systematic reliance on data. This means that a study, at least in principle and ideally in practice, could be replicated by another scholar. The idea of replication is essential to the project. First, the design of the study must be transparent so that anyone can understand the choices that were made in the research process. As you will see, many of these choices have a huge impact on the findings a study produces. Second, the possibility of replication removes—as much as possible—the biases of the researcher from the conclusions ultimately drawn. With transparent choices about the research process and a protocol that could be followed by someone else, any researcher working on the same project with the same research design should—theoretically—reach the same conclusions. Finally, replication allows a scholar to build on the work of others. Professors Epstein and King call quantitative methods of this sort a "social enterprise." Many of the studies we discuss in this book build on the previous work of others. For Discussion Do you identify more with the "qualifiers" or the "quantifiers"? What types of questions can be best answered by the "qualifiers"? What about the "quantifiers? Do you have an instinct about this? Do you think that what judges say in their opinions, or their off-the-bench writings and interviews, is inconsequential? 15 Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. Law Rev. 19-20 (2002). 10 II. IDEOLOGY Recall the attitudinal model of judicial decisionmaking described in in the Introduction to Part I. The model focuses on the Supreme Court, and "holds that justices make decisions by considering the facts of the case in light of their ideological attitudes and values. . . . Attitudinalists argue that because legal rules governing decision making . . . in cases that come before the Court do not limit discretion; because the justices need not respond to public opinion, Congress, or the President; and because the Supreme Court is the court of last resort, the justices . . . may freely implement their personal policy preferences as the attitudinal model specifies."16 The attitudinal model has had an enormous influence on the empirical study of judging. The model's core claim—that the Justices' "ideological attitudes and values" are the primary determinants of their decisions—has served as the hypothesis for countless studies. Although the attitudinal model is limited to the Supreme Court, empirical scholars have applied its insights to study the lower courts as well. This Part provides an overview of those studies, working carefully through them to expose the potential pitfalls and limitations, as well as the strengths, of this line of research. A. Constructing a Study: The Basics Let's start by focusing more closely on Professor Revesz's study of the D.C. Circuit's environmental cases in order to learn the basic parts of an empirical study. As we'll see in the following sections, each of these moving parts can have its own difficulties, both generally and especially in the context of trying to decide if the ideology of the judge matters. 1. Defining the Hypothesis One cannot begin an empirical study without an hypothesis: a statement of what the researcher hopes to prove or disprove. Professor Revesz's intuition was that judicial ideology affected the disposition of the cases in his dataset. But to get started, Professor Revesz wrote down his intuitions as hypotheses—theoretically motivated predictions of what he might find. One of Revesz's hypotheses was "selective deference" i.e., that "as a result of different views on the value of environmental protection, Democrats defer to the EPA when the challenger seeks laxer standards and Republicans defer more when the challenger seeks more stringent standards."17 In other words, conservative Republicans 16 SCAMR 110-11. 17 'Revesz, Environmental Regulation, Ideology, and the D.C. Circuit, supra, at 1728. 11 would tend to side more often than not with industry, while liberal Democrats would side with environmental interest groups. 2. Data Any empirical study needs data. You've heard the expression "garbage in, garbage out?" That's a reference to the fact that if you have bad data, you cannot count on the reliability of your conclusions. Empiricists either have to create a dataset or—as is commonly the case, because creating a dataset is extremely time-consuming and labor intensive—find one that is readily available. Because Professor Revesz was trying to pinpoint some of the factors that affect judicial decisionmaking, he needed a dataset that contained judicial decisions, with information about the judges that decided them. We've seen that he chose the D.C. Circuit for a reason. Decisions from the D.C. Circuit currently are published in the Federal Reporter, which one could find in any law library. Like most researchers today, however, Professor Revesz turned to electronic databases like Lexis or Westlaw, which contain all the information in the reporters and more. Using these databases, one can specify the years and the court of interest. Unfortunately, neither the Federal Reporter nor the online systems provides a definitive list of cases where a substantive environmental policy enacted by EPA regulation is being challenged. So what did Professor Revesz do? He used a series of searches in Lexis and Westlaw to produce a list of all cases where someone might be challenging the substantive environmental policy enacted by the EPA. He then had to look at the cases to sort the relevant cases from the irrelevant ones. Once that was done, Professor Revesz had a pile of cases to analyze. Most were decided by a typical three-judge appellate panel, drawn at random from all the judges on the D.C. Circuit, and a much smaller fraction were decided by the all the judges of the circuit sitting together, or en banc. (At the time Revesz was studying, the en banc court comprised between seven and nine judges.) 3. Independent and Dependent Variables Like any methodology, quantitative investigation involves some of its own terminology. The study is made using data (about this, more in just a moment), and the data is comprised of variables. Variables are the factors that are explaining or being explained in the data. There was something that Professor Revesz was trying to explain: why were environmental cases decided the way they were? And there was something that he thought explained the cases: the ideology of the judge. Empiricists call the thing they want to explain the dependent variable and the thing that does the explaining the independent variable. These are terms you will want to learn, and quickly. One way to remember them is this: the dependent variable depends 12 on the independent variable. As shown in Table XX, in Revesz's study the independent variable was the ideology of the judge; the dependent variable was the decision the judge reached in the case. In every empirical study, it is important to think carefully about what the dependent and independent variables are. Some studies have multiple dependent variables; others have multiple independent variables.18 Independent Variable Dependent Variable Definition The thing that does the explaining -> The thing we want to explain Revesz 's Study Ideology The decision reached by the judge Table XX. Dependent and independent variables. In order to make any progress, Revesz needed to specify his dependent and independent variables. Professor Revesz's dependent variable was whether the judge voted to affirm or reverse the EPA's decision in the face of a challenge by an industry group (which sought to limit environmental regulation) or a challenge by an environmental group (which sought to increase environmental protection). The independent variable was somewhat trickier. Like many other researchers, Revesz chose to use the political party of the President who appointed the judge as a proxy for the judge's ideology. His assumption was that Republican presidents are more likely to appoint a pro-business judge who is ideologically conservative rather than an environmentally oriented liberal. Thus, Revesz coded Republican appointees as conservative on environmental issues, and Democratic appointees as liberal. For Discussion Stop and think about Revesz' study. In a moment we are going to look at aspects of it, to both to see the challenges of empirical study generally, and to learn about ideology and judging. Before we dig in though, do you have any insight or intutions into things that one might critique in Revesz' study or hope to improve upon in future work? Recall the Epstein and King point about empiricism being a social endeavor. Anyone who delves into any empirical literature quickly learns that it is a world of small and slow improvements. Someone has a hypothesis. She builds a dataset and tests it. Critics complain about aspects of the study: bad data, poor choices of variables, etc. The In fact, Revesz's study had additional variables that we omit here for the sake of simplicity, [cite] 13 next scholar tries to improve upon the work. Gradually the collective enterprise zeroes in on an answer. Empiricists, like all scholars, have to have thick skins: we expect our work to be questioned. Through questioning, we learn. B. Selection Bias in Studying Judicial Decisions: Dataset Problems Constructing a dataset is a lot of work. Many other studies use established datasets,, but as we've seen, Professor Revesz created his own. There are a number of things that can go wrong when constructing a database. Perhaps the biggest challenge when constructing or choosing a dataset is avoiding problems of selection bias. Suppose you were interested in a real estate in a town, and were wondering how many bathrooms on average were in each home. To investigate this you chose to look at just the largest homes in town and count the bathrooms. Perhaps you computed that for this set of homes the average number of bathrooms is 3.2. Does this mean that the average number of bathrooms per house in the town is 3.2? Of course not. Looking just at the largest homes would bias the inferences we draw from the data. This type of bias is oftentimes called selection bias, as it stems from one's selection of data. When studying judging there are a number of different things that can cause selection bias. In the rest of this section we'll see two of them. (We'll see many more in later chapters on this book, especially Chapter XX that looks at the judicial hierarchy and the litigation process.) The practical effects of each cause of selection remain the same: the findings of the study might solely be due to the cases actually studied. 1. Published vs. Unpublished Opinions Judge Edwards takes Professor Revesz to task with regard to selection bias. Indeed, he writes: "A serious preliminary error of omission pervades the Revesz study, and the Cross and Tiller study as well. Both pieces assume that the only data relevant to the work of the D.C. Circuit are to be found in the published opinions of the court. As anyone familiar with the work of the court knows, literally hundreds of cases are decided each year without published opinion...The two studies take no account whatsoever of cases in which no published opinion was issued. . . The results in cases with no published opinion almost always produce unanimous judgments... The omission of cases of which the court disposes without opinions therefore materially skews the sample of the 'population' under analysis. The failure to consider these cases is a glaring mistake." At 1343. 14 His contention is that by ignoring unpublished opinions, Professor Revesz is getting an overly political and, thus, distorted view of the D.C. Circuit. Judges regularly designate some opinions as "publishable" and others as "unpublished." Before computers were widely used for legal research, various publishing companies would collect sets of decisions, sort them by venue or subject matter, and publish them as volumes. The cases that were published provided the set of precedents that could be cited by lawyers in future disputes. But some cases are disposed of without opinion. There are written opinions, but they are not published because they are deemed unimportant or insignificant in terms of adding something new to the corpus of legal precedents. Each court has its own set of rules about whether "unpublished" decisions may be cited, in part because those opinions were not available to all on an equal basis. The designations are misnomers today given the prevalence of electronic data; both published and unpublished opinions are available on on-line service such as Lexis/Nexis and Westlaw. But the distinction between published and unpublished cases is still an important signal of which opinions are ground breaking enough to be taken as precedents for later cases. And though there is some evidence of judges gaming this system, i.e. that they dispose of some cases in unpublished opinions when there was a novel issue at stake that would warrant publication, the point still remains. Judges don't publish all their decisions because many of them don't break any new ground. They are easy cases that, at most, involve the application of settled law to new facts. Indeed, the number of published cases is a substantial minority. Professor Pauline Kim and her coauthors report that only 41% of federal district court cases were briefed and submitted to a judge in 2006; of those cases, approximately 25% resulted in a published opinion.19 Judge Edwards' critique of Professor Revesz is theoretically persuasive on face, though in the case of Revesz' study [what about CrossTiller??] he simply got his facts wrong. Professor Revesz looked carefully for both published and unpublished opinions in his study. This is no easy task, as there is no single way to find all opinions. Professor Revesz performed many searches with a variety of parameters on Westlaw and Lexis/Nexis, and communicated with the D.C. Circuit clerk, to ensure that he had all of the cases. The Revesz study, thus, does not suffer from the ills of selection bias due to publication. But to see how looking only at published decisions can affect the inferences we can draw about judging, consider the study by Professor David S. Law, who looked at all Pauline T. Kim, Margo Schlanger, Christina L. Boyd & Andrew D. Martin, 29 Wash U.J.L. & Pol'y 83, 97 (2009). See also Evan J. Ringquist and Craig E. Em inert, Judicial Policymaking in Published and Unpublished Decisions: The Case of Environmental Civil Litigation, 52 POL. Res. Q. 7,9 (noting that "only 5 to 10 percent of all cases receive published decisions"). 15 asylum cases in the Ninth Circuit from 1992-2001. Asylum cases are a subset of immigration cases in which a person who otherwise is deportable argues they should be permitted to stay to avoid mistreatment by their home state. The U.S. Court of Appeals for the Ninth Circuit is the largest circuit court in the United States, with regard to population, size, the number of judges, and the amount of litigation. Because of its size and location in the western United States, the Ninth Circuit handles a disproportionate amount of asylum cases. The Ninth Circuit reviews asylum determinations made by the Department of Justice. As Professor Law explains, "the underlying legal question is whether the petitioner has demonstrated a 'well-founded fear of persecution' on political, ethnic, or religious grounds."21 This legal standard is ambiguous, which affords discretion to the judges deciding these cases. Petition Denied Petition Granted Withholding Granted 100 - 80 43 60 c O o o O Democrats Republicans Figure XX. Percent liberalism in published civil liberalism decisions in the U.S. District Courts, 1960-1976. N=13,326. Data obtained and figure adapted from C. K. Rowland and Robert A. Carp, A Longitudinal Study of Party Effects on Federal District Court Policy Propensities, 24 Am. J. Pol. Sci. 297 (1980). In the second study, Professor Orley Ashenfelter and his colleagues Professors Theodore Eisenberg and Stewart Schwab look at all federal civil rights and prisoner cases decided in the Central District of California, the Eastern District of Pennsylvania, and the Northern District of Georgia in fiscal year 1981.24 They looked at substantive final dispositions (that is, published or unpublished decisions ruling in favor of the plaintiff or the defendant), as well as settlements and procedural rulings (whether the case was referred to a magistrate, whether the plaintiff was represented by counsel, and whether any discovery occurred). Professor Ashenfelter and his colleagues looked at many the usual suspects to explain variation among the votes of the judges, including the political party of the appointing president, prior professional experience, age, sex, and religion. The results from their models are striking: Orley Ashenfelter, et al., Politics and the Judiciary: The Influence of Judicial Background on Case Outcomes. 24 J. Legal Stud. 259 (1995). 19 We find that judges influence the procedures within civil rights cases but have relatively little effect on whether cases settle or win. Further, judicial characteristics such as political party cannot explain what few effects we see. Many will be surprised that we cannot find that Republican judges differ from Democratic judges in their treatment of civil rights cases. The religion and gender of the judge had larger but still modest effects. One can always question the data or model, and we are careful not to accept our null hypothesis that there are no differences. But their failure to emerge in a reasonably sized fraction of all civil rights filings in a year (about 8 percent of the national total for nonprisoner cases) is evidence that individual judge characteristics cannot be assumed to influence substantially the mass of cases.25 In other words: Unlike the political science findings of ideological influence in published opinions, we find little evidence that judges differ in their decisions with respect to the mass of case outcomes. Characteristics of the judges or the political party of the judge's appointing president are not significant predictors of judicial decisions.26 The conclusion is that for this set of cases, the identity of the judge simply does not relate to the outcomes reached. FOR DISCUSSION 1. What do you think explains the pattern Professor Law found in published and unpublished asylum cases? Why might we expect a judge to be more likely to publish an opinion where asylum is granted rather than when it is denied? Is there a legal explanation? Or is something else going on? 2. Roland and Carp study different types of decisions than do Ashenfelder et al. Which differences do you think are most important? In which set of cases do you think there is more uncertainty about what the law is? 3. Professor Revesz is careful to note that the conclusions of his study do not apply to all litigation, but rather the cases that get disposed of by a court: The central purpose of this work is not to determine the probability that a challenger will prevail-a statistic that is likely to be biased by the possibility of settlement. Instead, it seeks to determine, from the Id. at 281. Id. at 257. 20 universe of litigated cases, whether differences in votes across judges can be explained by ideological factors.27 Would it be possible to learn about the likelihood of prevailing in a particular dispute by just looking at cases resolved in court? C. Ideology as Independent Variable: Proxy and Measurement Problems To determine whether ideology is doing the work—rather than something else— it's necessary to specify what ideology is, and find a way to measure it. What is ideology? Remember what Judge Cardozo said: "There is in each of us a stream of tendency, whether you choose to call it philosophy or not, which gives coherence and direction to thought and action."28 Ideology, in other words, consists of "normative commitments of various sorts."29 Or, as Judge Richard Posner and Professor William Landes put it, ideology is "a body of more or less coherent bedrock beliefs about social, economic, and political questions, or, more precisely perhaps, a worldview that shapes one's answers to those questions."30 Everyone has a bunch of normative commitments to all sorts of things. We are most interested in political ideology, meaning one's commitments to the sorts of public policy questions that feature in political debates and, inevitably, end up in court. In order to figure out to what extent ideology affects judging, we need to measure it. Judges and Justices rarely make public statements concerning their personal policy preferences. Unlike elected officials, they do not publicly associate themselves with one political party or another. How, then, should empirical scholars identify their ideology? As we saw in Chapter One, a common way of getting at any complicated variable is to use a "proxy," which is to say something that stands in for the thing we are trying to get at, but is easier to identify and measure. Of course, what we learn is only as good as the proxy itself. Finding proxies has been one of the most daunting tasks for scholars who want to study ideology and judging, and the use of these proxies is frequently under both challenge and improvement. (Recall, once again, the nature of empirical work as a social enterprise). Revesz, Environmental Regulation, Ideology, and the D.C. Circuit, at 1724. 28 Cardozo, supra, at 12-13. 29 Cass R. Sunstein, David Schkade & Lisa Michelle Ellman, Ideological Voting on Federal Courts of Appeals: A Preliminary Investigation, 90 VA. L. Rev. 301, 352 (2004). 30 Landes & Posner, Rational Judicial Behavior: A Statistical Study (cited in Edw/Liv pl920) 21 1. The Genesis of the Empirical Study of Ideology: Herman Pritchett and the Roosevelt Court We begin our discussion of proxies for ideology with the work of Herman Pritchett, who was the pioneer of the quantitative study of judging. A political scientist at the University of Chicago, Pritchett took the insights of the legal realists seriously. Pritichett wrote in the aftermath of the New Deal, a time when the Supreme Court was extremely prominent in American politics, largely because it kept thwarting President Franklin Delano Roosevelt's efforts to deal with the Great Depression. By the time Roosevelt was sworn in as President on March 4, 1933, the nation had been suffering from the Great Depression for over three years. Roosevelt's solution was the New Deal— a large package of legislative programs designed to effect economic recovery. Much of this legislation was quickly passed by Congress and signed by the President. However, despite the economic crisis at the time, the Supreme Court did not receive the New Deal legislation favorably. The first significant New Deal federal case that the Court decided was Schechter v. United States, 295 U.S. 495 (1935), in which the Court unanimously ruled that the National Industrial Recovery Act was unconstitutional because it delegated too much legislative power to the President, and because the poultry industry was essentially intrastate, not interstate, commerce, and, thus, could not be regulated by Congress. This was followed by a sequence of other cases in 1935 and 1936 in which a majority of the Court declared New Deal legislation unconstitutional, including the Agricultural Adjustment Act of 1933 in United States v. Butler, 297 U.S. 1 (1936), and a state minimum wage law in Morehead v. New York ex rel. Tipaldo, 290 U.S. 587 (1936). Roosevelt was a Democrat. At the time he took office, the Supreme Court was dominated by Justices who had been appointed by Republican Presidents: There were seven Republican appointees, and two really old Democratic appointees. See Table XX for the list of these Justices. The Court was often split on the New Deal decisions. Justices Stone, Brandeis, and Cardozo would oftentimes dissent, sometimes with Chief Justice Hughes. But the majority remained steadfastly opposed to the New Deal programs. 22 Justice Appointing President Year President's Party FDR's Replacement Justice Year Cardozo, Benjamin Hoover 1932 R Frankfurter, Felix 1939 Stone, Harlan Coolidge 1925 R Jackson, Robert 1941 Brandeis, Louis Wilson 1916 D Douglas, William 1939 Hughes, Charles Evans Hoover 1930 R Stone (Chief Justice) 1941 Roberts, Owen Hoover 1930 R Van Devanter, Willis Taft 1911 R Black, Hugo 1937 Sutherland, George Harding 1922 R Reed, Stanley 1938 Butler, Pierce Harding 1923 R Murphy, Frank 1940 McReynolds, James Wilson 1914 D Byrnes, James 1941 Table 1: Justices of the United States Supreme Court at the time of the inauguration of Franklin Delano Roosevelt and his appointees. Source: Lee Epstein et al., The Supreme Court Compendium (4th ed. 2006). The Justices are arranged based on the ideology of the seat's occupant in 1937, with more liberal Justices at the top of the list, according to estimates provided by Daniel E. Ho & Kevin M. Quinn, Did a Switch in Time Save Nine? 2 J. Legal Analysis 69 (2010). Roosevelt was stymied: during his first four years in office, he had no opportunities to change the membership of the Supreme Court. By 1937, the public outrage at the Supreme Court came to a head, and Roosevelt settled on a bold solution.31 He proposed a "Court-packing" plan that would allow him to expand the size of the Supreme Court to obtain a supportive majority. But during the country's heated consideration of the plan, Justice Roberts switched his vote in two New Deal cases, upholding the same sort of state and federal laws that he had voted to overturn the 31 See Barry Friedman, The Will of the People: How Public Opinion Has Influenced the Supreme Court and shaped the Meaning of the Constitution, Farrar, Straus and Giroux (2009). 23 previous year. There is great academic debate about what caused this "switch in time that saved nine," most of it focused on whether the threat to the Court caused a change in direction (a topic we take up in Chapter XI).33 What is unequivocal is that the Court started upholding New Deal measures without any changes in personnel. Eventually, retirements allowed President Roosevelt to appoint new Justices to the Court without adding new seats. Over the next three years, Roosevelt was able to secure a supportive majority on the Court. (Ultimately he would appoint 8 of its 9 members, as shown in the second column of justices in Table 1.) He began to shape the Court by appointing Justice Black. This appointment was followed by the appointment of Justice Reed in 1938, Justices Frankfurter and Douglas in 1939, and Justice Murphy in 1940. Once Roosevelt had a solid majority on the Court, however, an odd thing happened. You might have expected that we would see a great deal of agreement on the Court. After all, a majority of the Justices were appointed by Democrats, and they seemed to share similar views on the major issues of the day. But just the opposite happened. The Justices fragmented, leading to one of the highest dissent rates in history. As we will learn in the chapter on collegial court decisionmaking, this Figure is a little deceptive. It turns out - though this was not understood publicly until years later -that prior to some point in the 1920s, the Justices would often suppress their dissent and join majority opinions with which they did not agree. Once that "norm of consensus" shattered, the dissent rate started to pick up. But even yet, one would not expect such a sharp spike in the late 1930s and early 1940s, when the Court, ostensibly, was full of simpatico Justices. What explained this fragmentation? With Roosevelt's majority came a shift in focus by the Supreme Court. Where previously the Court's big cases had been in the area In West Coast Hotel v. Parrish, 300 U.S. 379 (1937), Justice Roberts joined the majority to uphold the constitutionality of a minimum wage law enacted in the State of Washington. Just one year earlier, with precisely the same Court membership, the Court had overturned a similar state minimum wage law in Morehead v. New York ex rel. Tipaldo, 298 U.S. 587 (1936), citing the sanctity of the individual liberty of contract. The Court had also held in Carter v. Carter Coal Company, 298 U.S. 238 (1936), that a congressional act which established a commission to determine fair wages and labor standards for coal miners was unconstitutional because the commerce clause does not permit federal regulation of production or manufacturing. However, eleven months later the Court found the National Labor Relations Act constitutional, finding in NLRB v. Jones & Laughlin Steel Corp., 301 U.S. 1 (1937), that activity which has a close and substantial relationship to interstate commerce can be regulated by Congress. 33 CITE: Daniel E. Ho & Kevin M. Quinn, Did a Switch in Time Save Nine? 2 J. Legal Analysis 69 (2010). See also Bruce Ackerman, We the People: Transformations, Cambridge MA: Harvard University Press (1998) and Barry Cushman, Rethinking the New Deal Court, New York: Oxford University Press (1998). 24 of economic regulation - and sometimes the Court struck down major statutes - now the emphasis gradually moved to civil liberties. In economic cases, the Roosevelt Court displayed a great deal of unanimity, almost always upholding what the legislature had done. But these same Justices frequently split on questions of civil liberties. Professor Pritchett was curious to understand what accounted for the Court's fragmentation. So, he decided to look at the behavior of the Justices of the Supreme Court during the 1939 and 1940 terms.34 (The Supreme Court sits in terms that begin on the first Monday in October and last through the first Monday in October the following year. The Court conducts oral arguments and announces decisions from start of the term until, typically, the end of June or early July.) Pritchett's hypothesis foreshadowed the later work of the attitudinalists: [N]o one doubts that many judicial determinations are made on some basis other than the application of settled rules to the facts, or that justices of the United States Supreme Court, in deciding controversial cases involving important issues of public policy, are influenced by biases and philosophies of government, by "inarticulate major premises," which to a large degree predetermine the position they will take on a given question. Private attitudes, in other words, become public law. More precisely, it is the private attitudes of the majority of the Court which become public law.35 Pritchett did not contend that the Justices' "private attitudes" determined their votes in all cases. He acknowledged that legal factors might drive decisions in cases where the Justices are unanimous: in such cases, he reasoned, "presumably the facts and the law are so clear that no opportunity is allowed for the autobiographies of the justices to lead them to opposing conclusions."36 On the other hand, as we saw in Chapter Two, law is sometimes indeterminate: In a substantial number of cases, however, the nine members of the Court are not able to see eye to eye on the issues involved. Working with an identical set of facts, and with roughly comparable training in the law, they come to different conclusions. If our thesis is correct, these divisions of opinion grow out of the conscious or unconscious preferences and prejudices of the justices, and an examination of these C. Herman Pritchett, Divisions of Opinion Among Justices of the U.S. Supreme Court, 1939-1941, 35 Am. Pol. Sci. Rev. 894 (1941) [hereinafter Pritchett, Divisions of Opinion]. 35 Pritchett, Divisions of Opinion, supra, at 890. 36 Id. at 890. 25 disagreements should afford an interesting approach to the problem of judicial motivation31 The Justices' practice of publicly dissenting from decisions of the Court in cases of disagreement gave Pritchett fodder for his study: A nonunanimous opinion admits the public to the Supreme Court's inner sanctum. In such a case the process of deliberation has failed to produce a conclusion satisfactory to all participants. Having carried the argument as far as they usefully can, the justices find it necessary finally to take a vote, state and support the winning and losing positions, and place arguments before the world for judgment. In informing the public of their divisions and reasons, the justices also supply information about their attitudes and their values which is available in no other way.38 So then, Pritchett developed a methodology that in retrospect seems trivially simple, but at the time was revolutionary. Rather than reading sequences of cases to understand what the law is - what the "qualifiers" urge - Pritchett collected every case decided during the 1939 and 1940 Terms and simply counted the number of times that the Justices agreed with one another. Table XX contains the chart he came up with (modified slightly by us). During these two Terms, there were eighty-nine "controversial cases"; i.e., cases in which there was at least one dissent. The percentage in each cell of this table represents the percentage of cases where the Justice on the row agreed with the Justice on the column. (The cells on the main diagonal are not reported because each Justice, obviously, agreed with himself in each opinion.) The ordering of the Justices is chosen to highlight the blocks of Justices who are in agreement. Id. (emphasis added). C. Herman Pritchett, The Roosevelt Court: A Study in Judicial Politics and Values xii.) (1948). 26 Justice McReynolds Roberts Hughes Stone -a CD CD Frankfurter* Murphy* M o cö 3 Douglas* McReynolds — 64 64 41 35 31 38 24 24 Roberts 64 — 75 51 45 45 39 37 36 Hughes 64 75 — 78 63 64 53 49 49 Stone 41 51 78 — 81 84 75 69 68 Reed1 35 45 63 81 — 86 80 79 79 Frankfurter* 31 45 64 84 86 — 91 85 84 Murphy* 38 39 53 75 80 91 — 89 89 Black* 24 37 49 69 79 85 89 — 100 Douglas* 24 36 49 68 79 84 89 100 — Table 2. Percentage agreements among Supreme Court justices in non-unanimous decisions for the 1939 and 1940 terms. Eighty-nine total cases. Adapted from Pritchett, Divisions of Opinion, supra. *Denotes Franklin Delano Roosevelt appointee. Does the chart tell you anything? Take the time to look at the data yourself and see if you can discern a pattern. We can help by adding a couple of perpendicular bold lines: 27 Justice McReynolds Roberts Hughes Stone -a CD CD Frankfurter1 Murphy1 M o cö 3 Douglas1 McReynolds — 64 64 41 35 31 38 24 24 Roberts 64 — 75 51 45 45 39 37 36 Hughes 64 75 — 78 63 64 53 49 49 Stone 41 51 78 — 81 84 75 69 68 Reed1 35 45 63 81 — 86 80 79 79 Frankfurter1 31 45 64 84 86 — 91 85 84 Murphy1 38 39 53 75 80 91 — 89 89 Black1 24 37 49 69 79 85 89 — 100 Douglas1 24 36 49 68 79 84 89 100 — Table 3. Percentage agreements among Supreme Court justices in non-unanimous decisions for the 1939 and 1940 terms. Eighty-nine total cases. Adapted from Pritchett, Divisions of Opinion, supra. 1Denotes Franklin Delano Roosevelt appointee. Do you see what the black lines show? There seem to be two sets of Justices that agree with one another at very high rates. The Justices in the upper-left quadrants and the lower-right quadrants tend to stick together. As Pritchett put it: "The table appears to reveal a marked division of the justices into two wings or groups. The first is composed of McReynolds, Roberts, Hughes and Stone; the other includes Murphy, Frankfurter, Black, and Douglas." What do you know about the people in the groups? Do you think it is a coincidence that they correspond to those Justices appointed by President Roosevelt, and those appointed by earlier Presidents? 28 Pritchett concluded that these patterns are explained by the ideologies of the Justices, with the Roosevelt appointees representing a left-leaning liberal coalition, and the remaining Justices representing a right-leaning, anti-New Deal minority. Pritchett himself called the two groups "right-wing" and "left-wing," and he argued that the Justices' "right-wing" and "left-wing" views explained the outcomes in a variety of issues dealt with by the Supreme Court: "It is not contended, of course, that the decisions were motivated wholly by the personal views of the justices, but the data clearly indicate that these views had a considerable effect in the process of making up the judicial mind."39 Now note: Pritchett talked about "ring-wing" and "left-wing" Justices, but how did he know who was a "right-winger" and who was a "left-winger?" The analysis itself only shows there are clumps of Justices; it doesn't tell us what caused the Justices to clump as they did. To make the leap from agreement to ideology, Pritchett did two things. First, he read the cases, and found that a majority of them split on the pro-government/anti-government or labor/business axes that structured the politics of the day. The groups of Justices that Pritchett identified as "wings" tended to cluster on one side or the other of these issues. The "left-wing" Roosevelt appointees usually found for the government or labor; the remaining "right-wing" Justices opposed the New Deal government policies or sided with business. Second, Pritchett assumed that the divisions on the Court resulted from differences of opinion as to desirable public policy; i.e., that the groups reflected relative "liberalism" and "conservatism" as those terms are understood by the man in the street. Pritchett, Divisions of Opinion, supra, at 897. 29 Pritchett's methodology was both simple and revealing. And it made sense to anyone who knew the Court of the day. We can do precisely the same thing with a more contemporary Supreme Court, and we suspect you will have the same reaction: Justice Thomas Scalia Alito Roberts Kennedy Breyer Souter Ginsburg Stevens Thomas* — 85 73 75 60 27 30 27 19 Scalia* 85 — 76 82 65 34 36 34 27 Alito* 73 76 — 86 76 42 36 35 28 Roberts* 75 82 86 — 74 44 42 37 32 Kennedy* 60 65 76 74 — 57 51 49 42 Breyer 27 34 43 44 57 — 73 71 72 Souter* 30 36 36 42 51 73 — 76 74 Ginsburg 27 34 35 37 49 71 76 — 75 Stevens* 19 27 28 32 42 72 74 75 — Table 4. Percentage agreements among Supreme Court justices in all non-unanimous decisions for the 2005-2008 terms (excluding Justice O'Connor, who participated in twenty-six cases during the 2005 term). Data obtained from http://supremecourtdatabase.org. The case citation is the unit of analysis. Not surprisingly, we see a very similar pattern. There is a bloc of Justices, including Justices Thomas, Scalia, Alito, Roberts, and Kennedy (all appointed by Republicans), who agree with each other quite frequently. There is another bloc, consisting of Justices Breyer, Souter, Ginsburg, and Stevens, who are also quite cohesive. Ideology might account for this pattern of behavior in this more modern Supreme Court. 30 FOR DISCUSSION 1. Suppose that during a particular term, the Justices chose how to vote on each case by flipping a coin: heads meant a vote for the petitioner (the party challenging the decision from the court of appeals), and tails meant a vote for the respondent (the party who won below). Here, politics, interpretive method, demographics, or political party would have no effect on decisions. What patterns of dissent might we expect to see if we were to conduct an analysis like that in Table 2 of this hypothetical Court? If random chance alone accounted for behavior, we would expect to see percentage agreements that were about the same for each pair of Justices. Each Justice's votes should split roughly 50-50 in favor of petitioner, or in favor of respondent. And, if each Justice flipped her own coin, each Justice should end up agreeing with every other Justice about half the time. Pritchett's findings are suggestive precisely because the groupings he found appear not to be the product of random chance. 2. In many cases, sorting Justices into a majority or minority coalition is straightforward. However, sometimes Justices publish concurrences, where one Justice agrees with the others on the disposition of the case (who wins and who loses), but does not agree with the rationale. How do you suppose Pritchett dealt with these situations? Even more rarely, a Justice will concur in part and dissent in part. How should a researcher assign a Justice to a majority or minority coalition in that case? 3. Professor Pritchett explicitly excluded unanimous opinions from him analysis. How might that choice skew the picture we get of the Supreme Court? To pursue this further, in Table 5 we present the Roberts Court analysis including unanimous opinions. During this time period, where the Court decided some highly controversial cases, over 42% of all cases were decided unanimously. 31 How does this table differ from the other Roberts Court table? Does this paint a different picture of what is happening on the Supreme Court? Justice Thomas Scalia Alito Roberts Kennedy Breyer Souter Ginsburg Stevens Thomas1 — 92 84 86 77 58 60 58 53 Scalia1 92 — 86 90 80 63 63 62 58 Alito1 84 86 — 91 86 65 60 60 56 Roberts1 86 90 91 — 85 68 67 64 61 Kennedy1 77 80 86 85 — 76 72 71 66 Breyer 58 63 65 68 76 — 85 83 84 Souter1 60 63 60 67 72 85 — 86 85 Ginsburg 58 62 60 64 71 83 86 — 85 Stevens1 53 58 56 61 66 84 85 85 — Table 5. Percentage agreements among Supreme Court justices in all decisions for the 2005-2008 terms (excluding Justice O'Connor, who participated in twenty-six cases during the 2005 term). Data obtained from http://supremecourtdatabase.org. The case citation is the unit of analysis. 4. How well would Pritchett's methodology work at studying the U.S. Courts of Appeals? What about a state court of last resort? A federal or state trial court? Can you see any difficulties with it? 32