A series of reports by the Transactional Access Records Clearinghouse (TRAC) and others — all based on case-by-case data from the Executive Office for Immigration Review (EOIR) — have found extensive disparities in how the nation's Immigration Judges decide the thousands of individual requests for asylum that they process each year. The consistency of these findings, as well as the fact that the disparities are found in most parts of the country and for individuals coming from many different nations, established that the background and experiences of individual Immigration Judges often are more important in how they decide a matter than the underlying facts.

This report card — following up on TRAC's 2006 and 2007 studies — is different. Its purpose is to see if the level of asylum disparity has changed. The study compares the asylum disparity rates that existed both before and after Attorney General Alberto Gonzales announced a Bush Administration plan in 2006 to improve the operations of the court. Accompanying this report are over 250 detailed reports providing the year-by-year record of individual Immigration Judges.

The central finding of TRAC's new report is that the judge-by-judge asylum disparities in the Immigration Courts are down. Among the fifteen immigration districts that decide the bulk of all asylum matters, the court data showed that disparity rates in ten of them have declined. This change in a significant number of busy districts does not mean that disparities have gone away. There were in fact, some districts where the disparities increased. But these findings indicate that the size of disparities have generally come down in magnitude. For example:

• Immigration Judges in New York City currently decide one out of every four asylum cases in the United States. In the three years before the AG's directive, the judge-by-judge differences in asylum decisions ranged from one judge who denied almost all (88%) matters to another who denied almost none (7%). In the last few years, this judge-by-judge variation was reduced by almost a quarter.
• In Miami, the second most active court, the reduction in judge-by-judge decision disparity was even larger. There the range among judges in their denial rates dropped almost two thirds from their previous levels.

TRAC found that the general improvement did not extend to every court. In fact, the data show that in a few Immigration Courts — for example in Cleveland and Orlando — the disparities had increased.

Background

In July 2006, TRAC published its first report on the handling of asylum cases by the Immigration Courts. After examining all recorded cases in which Immigration Judges decided asylum cases from FY 1994 through the first few months of fiscal year 2005, TRAC found wide disparities in asylum denial rates had persisted from judge to judge throughout this period.

As is the case in our current report, TRAC's 2006 report included a series of focused studies that examined the work of more than 200 individual judges who had decided at least 100 asylum cases. For each one, the report provided specifics on the asylum cases handled by that judge, information about the judge's background, and statistics comparing that judge's asylum denial rate with his or her colleagues in the same city, and throughout the rest of the United States.

In August 2006, shortly after TRAC's report was published, the AG directed EOIR "to review this study and provide an analysis and, if appropriate, recommendations to the Deputy AG with respect to this issue."

EOIR did undertake this review. And in response to a Freedom of Information Act request, it recently provided us the text of this internal study. Interestingly enough the EOIR questioned the value of using statistics to compare judge-by-judge denial rates. In the conclusion of the analysis, for example, the agency said "it is very difficult if not impossible, for anyone to provide a definitive, well-supported comparison of the denial rates of the more than 200 judges in the immigration court system." The review also found that "most judges appear to grant and deny asylum claims at comparable rates."

The EOIR also noted, however, that "there are some notable variances in denial rates . . . attributable to a relatively small number of judges," In summary:

"[S]ome of the largest disparities in the denial of asylum claims among a handful of immigration judges merit close attention. EOIR is committed to addressing this issue through an enhanced training, mentoring, and supervisory program that will not improperly impinge upon the adjudicatory role of the immigration judges, but that will ensure that all immigration decisions are based on the law and facts in each specific case."

The broad view that the disparities in the Immigration courts were not susceptible to statistical analysis was not shared by the General Accounting Office. In September of 2008 it published an extensive report on the subject, based on records about all asylum decisions made by the courts from October 1, 1994 to April 30, 2007. In its study disagreeing with the EOIR the GAO concluded that "[t]he likelihood of being granted asylum varied considerably across immigration courts and judges," and persisted even after taking into consideration a long list of other factors related to decision outcome.

In a very recent submission to Congress in May — mandated by a provision in the Omnibus Appropriations Act of 2009 — the EOIR discussed its implementation of the various changes outlined in 2006 by the AG. Concerning the disparity problem, the EOIR said:

"EOIR continues to monitor asylum grant rates and closely supervises immigration judges who have unusually high or low asylum grant rates. In addition, EOIR provides immigration judges with targeted training, mentoring, peer observation, and counseling, as appropriate."

The EOIR's short report to Congress did not provide any assessment of the effectiveness of these actions. TRAC's analysis seeks to determine whether asylum decision disparity has changed since the AG's directive on improvements was announced.

Changing Disparity Levels Within the Immigration Courts

The study compares the asylum decision disparity levels within Immigration Courts before and after AG Alberto Gonzales announced a Bush Administration plan in 2006 to improve the operations of the court. By "disparity" we mean differences among judges' decisions that cannot be attributed to differences in the merits of the individual cases being decided.

Meaningful disparity assessments are possible because the Executive Office for Immigration Review assigns incoming asylum applications to judges on a random basis, using automated procedures within each court. When individual judges decide a significant number of asylum requests, this random assignment results in all judges dealing with the same broad mix of asylum seekers as their colleagues on that court. Therefore, differences in judge-to-judge denial rates for judges serving on the same court should not be attributable to differences in the merits of the individual cases heard by each judge, and thus can provide a way to assess "disparity" in that court's decisions.

Each judge who made at least 75 asylum decisions on the merits in a specific Immigration Court during either period was included in TRAC's analysis. These judges are listed by Immigration Court in Table 1. For each judge, Table 1 shows the total number of asylum decisions each judge made and the judge's denial rate in each of the two periods — before and after the AG's directive.

Judge-to-judge asylum decision disparity within a court can undergo change for many reasons. One reason is that some of the judges may have altered their decision-making practices over the two periods. An examination of Table 1 reveals that while many judges had quite similar denial rates in both periods, some judges for whatever reason had a somewhat lower or higher denial rate in the previous to as compared to the most recent period.

But within any court, shifts in judge-by-judge disparity patterns may also occur because there are different judges sitting on that court. As can be seen through an examination of Table 1, not all judges are listed for both time periods. During this six year period, 65 judges left the bench while 75 judges were hired (see TRAC's earlier report on judges). In addition, a judge might not be "new" to the bench, but have moved from one city to another. Such a judge will show up as "leaving" one court and being a "new" judge in the court to which they had transferred. There was also a small number of Immigration Judges who moved to (or from) an administrative position, or simply decided too few asylum cases in one period to be included. For simplicity in the analysis which follows, we refer to any judge listed who only shows a denial rate in the "before" period to be a judge who "leaves" and a judge who only shows data in the "after" period to be a "new" judge.

While there are more than 50 Immigration Courts around the country, and indeed during this period five new Immigration Courts joined their ranks, a much smaller number of courts handle most asylum cases that are filed. This analysis focuses on fifteen large Immigration Courts which accounted for more than four out of five decisions in asylum cases during the period. These fifteen courts were the only ones where there were four or more judges who had decided at least 75 asylum cases, thus allowing differences in their denial rates to be compared (see Table 1).

Because the majority of asylum cases are decided in only four of the nation's immigration courts — New York, Miami, Los Angeles and San Francisco — these are examined in greater detail below, followed by some observations on the remaining eleven courts.

New York: The judges on this court decided one out of every four asylum decisions in the country during the period covered in this study. In the three years before the AG's directive, as shown in Table 2, the range between the judge with the highest denial rate (88%) and the lowest (7%) was extremely wide — a difference of 81 percentage points. In the period after the AG's directive, the disparity markedly decreased. The new highest denial rate had come down to 67% while the lowest rate was 5% — a difference of 62 percentage points, or about a quarter less.

While this is one way to examine disparity — comparing the denial rates of the two judges who differed the most — they obviously were only 2 out of the 28 judges on the New York court deciding asylum cases. The study therefore looked at disparity a second way. Rather than focusing on the extremes, the study compared decision patterns for the middle fifty percent of the judges serving on the bench. By "middle half" we simply mean if the 28 judges were listed by denial rates from highest to lowest, we examined the 14 (half of 28) whose denial rates fell in the middle on this list. That is, there are 7 judges with higher denial rates, and 7 judges with lower denial rates. We then looked at how close or far apart were the denial rates for just these 14 middle judges. Statisticians refer to this as the middle or interquartile range — the difference between the 25th and 75th percentiles.

During the first period, before the AG's directive, the denial rates for these "middle" judges went from 23% at the low end to 55% at the high end — a range of 32 percentage points. By comparison, since FY 2006 the denial rates for these "middle" judges ranged between 14% and 34% — a much smaller spread of 20 percentage points (see Table 3).

Figure 1. Click for details.

How did these changes come about? Did the judges change their decision practices, or did the identity of judges serving on this court change? Figure 1 displays the distribution of each judge's denial rate in the "before" and in the "after" period. In the upper plot — concerning the "before" period — eleven judges had denial rates of 40 percent or higher, while in the lower plot concerning the "after" period, only four judges had denial rates that high. Part of the shift occurred because three judges with high denial rates left. The judges who left are distinguished by white diagonal stripes in the upper plot. But more of the reduction in the judge-to-judge decision disparity occurred because the judges who stayed moved closer to the denial rates of their colleagues. While considerable disparity continues to exist on the New York Immigration Court, the comparison plots show clearly that there has been a sizable reduction in the past two and a half year period.

Miami: The Miami Immigration Court is the second most active in the country, deciding one out of every six asylum cases. In this court during the three years before the AG's directive, denial rates ranged from a low of 25% to a high of 98% percent — a range of 73 percentage points. In the current period, the lowest denial rate was 65% and the highest rate 90.5% — a range of only 25 percentage points, or variability that is almost two-thirds smaller than before.

Once again, in addition to examining the polar extremes, TRAC looked at the denial rates of the middle fifty percent of the judges serving on the Miami Immigration Court who are listed in Table 1. In the three years before the AG's directive, their denial rates varied from a low of 78% to a high of 90% — just 12 percentage points. This variability is much less than was found among the immigration judges in New York. In the current period the range was 11 percentage points or virtually the same. These changes are summarized in Table 2 and Table 3.

Figure 2. Click for details.

A better picture of how a court's judge-to-judge disparity rate has shifted can be gained by comparing graphs of how the individual judge denial rates were distributed in the "before" versus "after" periods (see Figure 2). In the "before" period there were two judges who had denial rates markedly different from the rest. These two judges were not on the bench in Miami in the recent period, and thus we see much less of a difference between the lowest and highest judges' denial rates. However, we can also see that the number of judges at the other extreme — denying 90 percent or more of asylum applications — also came down. This occurred not because judges left the bench but because of a change in decision patterns. Indeed, there was a tendency for judges in the 80 to 90 percent denial range to have somewhat lower denial rates in the more recent period.

Los Angeles: As in New York and Miami, asylum decision disparities in Los Angeles were reduced, although the nature of the changes was somewhat different. In the three years before the AG's directive, judge denial rates ranged from a low of 33% to a high of 87% — a spread of 54 percentage points. Subsequent to the AG's directive, the range came down only slightly — from a low of 37% to a high of 84% — a spread of 47 percentage points. But TRAC's analysis of the judges not at the extremes found considerable movement. Before the AG's directive the middle half of LA judges ranged from a low of 47% denial rate to a high of 75%, a spread of 28 percentage points. After the directive, this variability was cut by more than half — down to a spread of only 12 percentage points (see Table 2 and Table 3).

Figure 3. Click for details.

These changes can be distinctly seen in the before and after plots for the Los Angeles Immigration Court in Figure 3. While the plot shows that the overall range in denial rates didn't change, a number of judges at either extreme of this continuum moved towards the middle, closer to each other, in denial rates.

This court also saw a sizeable turnover in judges serving on the bench. However, for the most part, judges who "left" and "new" judges were not concentrated at the extreme ends but distributed along the entire range and therefore did not account for the overall reduction in judge-to-judge disparity observed.

San Francisco: While there was some reduction in decision disparity among the judges on the San Francisco Immigration Court, the trends are less clear. In the three years before the AG's directive, judge denial rates ranged from a low of 33% to a high of 91% — a spread of 58 percentage points. In the current period, this spread between the highest and lowest had come down to 46 percentage points, with a low of 39.5% and the high of 85%. As shown in Table 3, there was also a slight reduction in the spread of the middle half of the judges on the San Francisco court — from a spread of 26% down to 22%.

Figure 4. Click for details.

Figure 4 provides a clearer view of the situation. In the top graph, it is clear that a significant number of judges in the upper half of the distribution (denial rates from 50% to 100%) are not represented in the bottom graph. For judges that remained, what appears to be occurring is not a movement towards a single central point but a larger group of judges moving toward each other at the lower end of the continuum, while a second smaller group is concentrating at the upper end. Judges on this court thus appear to be bifurcated in their asylum denial rates.

Additional Courts: As noted above, TRAC found that there were noticeable declines in the variation among judges in their asylum denial rates in ten out of the fifteen large Immigration Courts TRAC examined. The difference in denial rates between the judges with the lowest versus the highest denial rates declined in Arlington, Baltimore, Boston, Chicago, Newark, and San Diego Immigration Courts (see Table 2). Newark, for example, cut its disparity by about half due both to changes in the makeup of judges serving, as well as changes in the decision patterns of its judges. In Boston, the disparity was cut by over 60 percent. Among the remaining courts, however, only Newark showed any discernable reductions in the disparity in denial rates for judges who weren't at the extreme ends of the continuum (see Table 3).

In four courts — Cleveland, Houston, Orlando and Philadelphia — the range between the highest and lowest denial rates increased rather than decreased. In Orlando and Cleveland the disparity — already sizeable in the three years prior to the AG's directive — increased substantially. These two courts also experienced large turnover in their judges. See Table 1. The one remaining Immigration Court out of the fifteen examined — in Atlanta — showed little change.

Disparity in Outcomes for Asylum Seekers from Different Countries

TRAC also examined asylum denial rates by country. For each of the two periods — FY 2004 to FY 2006 and FY 2007 to FY 2009 — Table 4 presents the number of asylum decisions and the denial rate, by nationality of asylum seekers. Table 4B lists similar information ordered by denial rate, but whereas the Table 4 lists all countries, Table 4B includes only those countries that had at least 100 asylum decisions during this period.

Figure 5. Click for details.

Few nationalities had a sufficient number of cases to allow meaningful judge-by-judge comparisons between the two periods. Figure 5 displays the countries with the largest numbers of asylum seekers. During FY 2004 to 2006 China headed the list, followed by Colombia and Haiti. After these three countries, the number of asylum applicants by nationality drop quite rapidly. During the last two and a half years, the composition of asylum seekers by country underwent some changes. While China was still number one, Haiti had moved into second place followed by El Salvador. But in both periods individuals from only a few countries were dominant.

When asylum seekers by country are further divided by Immigration Court, there were only seven Immigration Courts which decided a sufficient number of cases from any single country to make analysis of decision disparities possible. Six of these court comparisons involved a single country, while for the seventh — Miami — it was possible to compare each of three different countries. Across these seven courts, a total of nine comparisons involving six nationalities could be made. The six nationalities were: China, Colombia, Ethiopia, Haiti, India and Venezuela. We compared judges if they had made at least 50 decisions for asylum seekers from any specific country. Again, TRAC's comparison required that there be at least four judges within a court who had decided a sufficient number of cases involving asylum seekers from that particular country.

The general decline in asylum disparity that was previously noted in many of the Immigration Courts also prevailed when asylum decisions were examined by nationality group. As shown in Table 5, the degree of judge-to-judge disparity declined in six out of the nine of these comparisons. In two there was little change, and in only one did we see increased decision disparity. The one court showing an increase was the Orlando Immigration Court. Orlando was also a court that had increased the level of decision disparity in its general asylum workload.

When we examined decision patterns for the middle fifty percent of the judges deciding cases on asylum seekers from a particular nationality group, declines were again seen. Seven out of nine comparisons showed some decline in decision disparity. Levels of change, however, were typically modest. Orlando again showed an increase in disparity, although its level of change was relatively small. See Table 6.

A Cautionary Conclusion

The goal of reducing disparities is not to develop a lockstep court system where all the judges in it are under pressure to decide the cases they are handling in the same way regardless of the facts. That is the very opposite of justice. Instead, the judges in all courts should constantly seek to recognize the valid claims of those who come before them and deny the claims that are legally insufficient.

"Equal Justice Under the Law," the idealistic goal carved into the mantle above the Supreme Court's main entrance, has been adopted in the mission statement of the Justice Department's Immigration Courts in its commitment to provide a "fair, expeditious, and uniform application of the nation's immigration laws in all cases."

Maintaining this standard — that similarly situated individuals will be treated in similar ways — is very demanding, particularly in an extraordinarily busy court like the EOIR, which sometimes has been subject to heavy-handed political manipulation. In an institution that for many years has steadfastly refused to even examine the evidence of extensive disparity contained in its own records, the challenge is even higher.

Despite a series of rigorous studies by TRAC, the Government Accountability Office, academics, newspapers like the San Jose Mercury News and others, EOIR managers have either denied the findings completely or perhaps sought to resolve them by pressuring the judges at the extremes to change their ways. When the court data from the last few years are compared with that of a more distant period, the evidence suggests that the long-ignored disparities of the court have declined. This is good. The goal, however, is a system that focuses on deciding correctly, not one that becomes obsessed with consistency.

There already are troubling signs, however, that EOIR's information recording practices may be partially hiding the true magnitude of the problems that beset the courts. This concern has arisen because the EOIR appears to have left it up to the Immigration Judges to decide whether their decisions to refuse asylum requests are recorded as a "denial" or as an "other completion."

What difference does this shift in recording practices make? After all, whether a decision is recorded as a denial or as an "other completion," the asylum application is not granted. TRAC has discovered, however, that since the AG's 2006 directive requiring (among other steps) that the EOIR study the disparity question, the number of "other completion" decisions jumped sharply. "Other completions" aren't counted in computing denial rates. And now, for the first time, "other completion" decisions for asylum cases exceed the number of denials.

The importance of this issue becomes clear when considering the EOIR's annual reports, in which supposedly accurate information is regularly presented to the public and Congress about the agency's performance in many different areas, including its grant and denial rates of asylum requests.

Specifically, recent EOIR reports claim its asylum denial rates are down. Indeed, in the last three years, the data do show the EOIR's nationwide asylum denial rates dropped from 60% down to 54%. TRAC found, however, in reanalyzing the EOIR data, that fully two-thirds of this so-called decline could be accounted for by the sharp increase in the use of the "other completion" category to record asylum outcomes. And it should be recognized that under the system, in many circumstances individual judges can reduce their denials simply by recording the decisions not to grant asylum in a different checkbox on a form. This means that Immigration Judges can appear less "extreme" in their denial rates simply by altering how they record their asylum decisions. And these recording changes in turn may have substantially altered our nationwide picture of how the asylum system is working.

EOIR's own internal analysis in 2006 in response to the AG's directive found a lack of clarity in practices used by judges when recording a decision not to grant asylum as a "denial" versus an "other completion." Now three years later even though EOIR has transferred to a newer and more up-to-date database tracking system, the agency has not taken steps to improve reporting standards so that actual asylum decisions are recorded in a more consistent and detailed manner.

EOIR's failure to take these simple practical steps to improve its recording and reporting practices raises serious questions about whether the agency has any real commitment to addressing the widespread and continuing problem of asylum decision disparity in this country.