Incomplete and Garbled Immigration Court Data Suggest Lack of Commitment to Accuracy
UPDATE: On November 4, 2019 TRAC sent a letter to the EOIR requesting a correction to public statements made on the agency's behalf regarding the quality of the data described in this report.
TRAC recently discovered gross irregularities in recent data releases from the Executive Office for Immigration Review (EOIR), the agency that oversees the US immigration court system. After attempting - unsuccessfully - to work with the EOIR to fix these problems, TRAC decided to make public our observations of the quality of the agency's public data releases as well as express our concerns about the lack of commitment within the agency to responsible data management.
Policymakers and the public routinely put their faith in federal agencies to provide complete and accurate information about their work. The value of government transparency is even higher in the area of immigration law and the Immigration Courts, which have become topics of considerable concern for Americans from all walks of life and for all three branches of government. In the present context, TRAC views concerns about EOIR's data inconsistencies - outlined below - as substantive, ongoing, and in need of prompt attention. Of greatest concern is the lack of commitment from EOIR to ensuring the public is provided with accurate and reliable data about the Court's operations.
"Significant Errors" in Past EOIR Data
This is not the first time the public has identified significant inaccuracies in EOIR's reported data. For instance, the Supreme Court of the United States relied upon figures provided by the EOIR as the basis for a major ruling affecting ICE detention practices. After the Supreme Court decided the case, the public discovered that the figures provided by the EOIR were fundamentally wrong. The EOIR did not uncover the data irregularities on its own. The EOIR's mistakes were only recognized because the public obtained the underlying data through a Freedom of Information Act (FOIA) request and identified the relevant discrepancies.
After the public alerted the government to its inaccuracies, in 2016 the U.S. Solicitor General was compelled to issue a formal letter to the Supreme Court apologizing for providing inaccurate data. The following excerpt of the Solicitor General's letter on August 26, 2016 attests to this error:
This example illustrates the very real danger posed by the EOIR's mishandling of data, as well as the value to society - and the government itself - of ongoing oversight through Freedom of Information Act (FOIA) requests. Despite the EOIR's past data mistakes, however, the quality of the agency's data releases has recently declined to unacceptable levels, as we discuss in the following section.
Recent Data Trouble at the EOIR
As a result of TRAC's ongoing FOIA requests, the Executive Office for Immigration Review releases a large batch of anonymized data about Immigration Court cases every month. Statistics on the operation of the Immigration Courts largely rely on information kept in a massive database maintained by the EOIR. The EOIR records information on each matter filed with the court and tracks subsequent events as the Court processes each case. This data is central to the Court's ability to efficiently and effectively manage its workload.
Although this data is highly valuable to policymakers and the public, the EOIR's mishandling of the data undermines its accuracy and public value. These data problems have been occurring with increasing regularity. Severe irregularities with the September 2019 data release set a new low.
On October 9, 2019, the EOIR responded to TRAC's FOIA request for updated case-by-case data through September 2019. TRAC promptly began processing the data in order to update TRAC's online tools and reports, and discovered serious inconsistencies that made the data unusable. TRAC alerted the EOIR to the problems we uncovered. The chronology below summarizes the cycle of data mishandling for the September data release, and TRAC's attempts to work with the EOIR to obtain complete and corrected data.
After this series of mistakes, TRAC urged the agency to implement basic quality control procedures to ensure that the EOIR's data releases to the public were not inadvertently garbled or incomplete. Moreover, TRAC expressed concern about the EOIR's underlying data management practices which posed a risk to both the public and the government if left unaddressed. We conveyed these concerns to EOIR noting specifically:
On Friday, October 25, 2019, while admitting mistakes had been made, the EOIR dug in its heels. The agency responded to TRAC's entreaties by sidestepping the underlying issue and avoiding responsibility for its routine inaccuracies:
TRAC was forced to take note of the EOIR's unwillingness to fully correct their mistakes and to work with the public to resolve the declining quality in their data releases.
The Case of the EOIR's Disappearing Data
After recognizing the seeming inability of EOIR to produce a correct and complete data release for September 2019, TRAC began digging deeper into the problem.
Our concern about EOIR's data was already heightened. We recently discovered that some months ago the EOIR had begun silently deleting swaths of records in their entirety from the data releases that we and other members of the public received. EOIR belatedly told us that withholding of entire records was necessary to protect immigrants' privacy. This rationale was perplexing since these records were already anonymized and all identifying details deleted. Regardless of the EOIR's justification for withholding the records, the agency had started making these deletions without alerting us that it was doing so, and failed to mark the data in any way to indicate the magnitude of the deletions or indicate in which files the deletions occurred.
TRAC is in a fairly unique position to examine this problem. For many years, TRAC has been regularly requesting snapshots of anonymized data from EOIR's database as part of our mission to provide the public (and often other government agencies themselves) with access to reliable, accurate data about the Immigration Courts. Because we receive and retain these monthly snapshots, we are able to monitor changes in these releases over time and assess whether releases are incomplete or inaccurate in some other way.
Therefore, TRAC undertook a careful comparison by matching the records received in the September 2019 release against the EOIR's release for the previous month, August 2019, and with the release we received a year ago for September 2018. This time we matched records based on unique identification numbers rather than simply comparing total record counts. This allowed us to identify records which the EOIR released in the past but were missing entirely from the current shipment.
The results of this comparison were sobering. Compared to the August 2019 release, the (allegedly-accurate) final September 2019 release was inexplicably missing more than 1,500 applications for relief that were present the previous month. We further found that 896,906 applications for relief which were present in the September 2018 release from a year ago were missing from the September 2019 files we received. This discrepancy of nearly a million records largely occurred because the EOIR appears to have started silently but systematically deleting records.
Compared to the data from August 2019, the EOIR's files for September were also missing records on over 600 charges DHS had filed. Also missing were over 700 case and/or court proceeding records, and over 900 records on scheduled hearings. An additional 1,200 records flagging various specific types of cases were also missing. For context, this flagging system is used to identify juveniles, recently arrived families seeking asylum, and immigrants required to remain in Mexico under the Migration Protection Protocols, and other special cases.
When the records in the September 2019 release TRAC received were matched with those from the September 2018 release a year earlier, the problems we uncovered multiplied. It was clear that the problem of missing records grew by leaps and bounds with the passage of time.
EOIR Data Management: Problems and Solutions
Based on the investigation above, TRAC identified key gaps in the EOIR's data verification procedures that lead to unreliable and inaccurate data releases.
If the EOIR does not implement basic data verification procedures, the public cannot tell if records were intentionally withheld and why they were withheld, or if records were accidentally omitted during the data copying process. The failure to address these problems also means that the public has no way to test for potential problem areas in EOIR's underlying master data files.
Accuracy, Reliability, Cooperation
Under any circumstance, maintaining a massive database of this nature is challenging. Clearly it requires the resources necessary for day-to-day operations. More fundamentally, however, it requires a commitment on behalf of the agency to provide the public with complete, accurate, and reliable data about the agency's operations. When TRAC uncovered unexplained data issues in the past, we have brought them to the attention of the EOIR and generally found the agency to be fairly responsive and committed to ensuring accurate reporting. The recent change in posture is therefore concerning. Moreover, because EOIR's data are relied upon as part of the official record of court filings and proceedings that have taken place, one should not expect official records to simply go missing without explanation.
It is deeply troubling that rather than working cooperatively with TRAC to clear up the reasons for these unexplained disappearances, the agency has decided to dig in its heels and insist the public is not entitled to have answers to why records are missing from the data EOIR releases to the public. TRAC urges the EOIR to take the basic steps necessary for managing any large database, especially a database of as inestimable value and relevance as the one EOIR maintains for the Immigration Courts.