The New York Times
October 18, 2017
By John Pfaff
A report released by ProPublica suggests that Supreme Court justices struggle with statistical data. Credit Gabriella Demczuk for The New York Times
Supreme Court justices have a tough job. They are required to hand down decisions that can affect millions of people and cost billions of dollars. And while some of the issues before them are purely legal, many turn on complex policy questions: Do black voters in the South still face substantial discrimination? How reliable are eyewitnesses? Are people convicted of sex offenses very likely to reoffend?
These questions can be answered only by understanding what the data says. Unfortunately, a report released by ProPublica on Tuesday suggests that the justices struggle with that task. Looking at a random sample of cases from 2011 to 2015, ProPublica found that the court cited faulty research or introduced its own errors in nearly a third of the 24 cases that relied on such facts.
In 2013, for example, Shelby County v. Holder invalidated a critical portion of the Voting Rights Act of 1965, making it arguably one of the most consequential cases in recent years. Justice John G. Roberts Jr., arguing that the South had taken great strides that made the protections of the act unnecessary, based his decision in part on a Senate Judiciary Committee analysis that misinterpreted how the Census Bureau reports race and ethnicity data and wrongly suggested that registration gaps between minorities and whites had shrunk significantly, an error that neither he nor his clerks caught.
As ProPublica itself acknowledges, its sample is too small to draw any solid statistical inferences, but the results are still troubling. They are also not particularly surprising. None of the justices has any serious training in statistics, and the clerks who assist them are almost all recent law school graduates, who rarely have any formal statistical background. Empirical facts are central to what the court does, but its members lack expertise.
Sometimes justices seem almost amused by that lack. When presented with potentially critical empirical evidence in a major gerrymandering case this month, Chief Justice Roberts joked that “it may be simply my educational background” before describing the material as “sociological gobbledygook.”
Policy is a major part of the court’s docket now, whether it likes it or not. The justices cannot avoid adapting to this, and they cannot simply dismiss evidence they don’t understand as “gobbledygook.”
The court has historically relied on amicus briefs, written by outside experts, to provide it with that broader empirical background and help compensate for its own institutional shortcomings. Unfortunately, these briefs are easily abused. In a 2014 article, Allison Orr Larsen, a law professor at William & Mary, pointed out that many amicus briefs include false or unsubstantiated empirical assertions, at least some of which make it into justices’ opinions. ProPublica similarly identified amicus-writing organizations that could not explain where specific numerical claims came from.
It would be easy to avoid the more egregious problems by simply refusing to rely on unsourced claims. But that fix would not go very far. It’s not hard to get factually dubious articles published. Some industry groups appear to have their own nominally peer-reviewed journals, which provide the illusion of respectability for results skewed to advance their interests. And major cases that make it to the Supreme Court often spend years in the trial and appellate stages, providing interested groups plenty of time to gin up favorable findings.
So what to do?
In the 1980s, the legal expert Kenneth Culp Davis proposed that the court create an outside research organization, akin to the Congressional Research Service, to do research on its behalf. However worthwhile, the idea went nowhere.
Perhaps a more viable idea is one that Mr. Davis rejected: establish a group of technical advisers to the court. A small team of social scientists and statisticians could help justices sift through empirical evidence. There is no shortage of scholars with Ph.D.s who would be eager to do that work for the court.
