New APPS

The judgments of algorithmic systems are often difficult to understand.  The system says you do not qualify for this loan – sorry!  You no longer qualify for x benefit – sorry! Why? Well, that’s just what it says.  And it’s impossibly opaque, for a variety of reasons – the underlying code and data are trade secrets, the decision is made by a neural network with millions of parameters, and so forth.  But suppose you object to the decision, or want to know how to do better?  This is the domain of, among other things, explainable AI (XAI).  XAI is normatively complicated, and I don’t want to get into that debate here.  Here I want to briefly look at a logically prior question: before we get to XAI, is there some sort of a right an explanation?  Is there a moral or legal sense in which the recipients of adverse algorithmic decisions are entitled to some sort of explanation of the basis for that decision in terms that they would understand and either find legitimate or be able to argue against? The particular point I want to make is that we need to think about the abstract right to explanation with reference to the reality of the U.S. legal system.

In a recent paper, Brett Karlan and Henrik Kugelberg argue against the need for a right to an explanation of AI decisions.  They take particular issue with an earlier paper by Kate Vredenburgh, which had argued for a right to an explanation on the grounds that it is necessary to fight back against algorithmic governance.  For the same reasons that the judge or bureaucrat owes you the reason for an adverse decision, so does the judge or bureaucrat using AI.  Using AI shouldn’t relieve them of the responsibility to inform people of the process that was used and how. She explains that “decision-makers are required to provide individuals with rule-based normative explanations and rule-based causal explanations … such explanations are necessary to enable individuals to engage in informed self-advocacy and are tolerably costly.” Normative explanations should say what the rules are and the normative reasons in favor of them; causal explanations are important for agency – they explain “what an agent would have to do to get a desired outcome, in terms of the relevant rules and robust population-level causal generalizations.”

It is not clear that either of these is actually possible in many cases (see, e.g., this paper by Andrés Páez (pp. 5-7) or this one by Boris Babic and I. Glenn Cohen), but leave that to the side.  Karlan and Kugelberg target the normative case.  As they summarize:

“We argue, first, that explanations are often either costly and practically impossible (as Taylor, 2024 warns) or else superfluous to the relevant normative good. Second, applying the right to an explanation to machine learning research (as well as other scientific research that uses machine learning) treats algorithmic decisions as normatively separate from our best approach to science policy more generally, a separation we argue is unmotivated. Finally, we argue that a right to an explanation leads to bad policy and bad design of bureaucratic institutions. In short, there is no reason to prefer a right to an explanation, and several harms that such a right imposes on us.”

On the last point, using climate change as an example, they point out that if we required that high-level policy be explainable in terms that the average person could understand, we’d have to abandon the use of anything more complicated than a linear regression in setting science policy.  That would make a mess of climate policy in particular, given the complexity of the models; the result would be policy that was inferior by pretty much every other measure.  Hence the right to an explanation would be “too costly” in such cases. This seems correct and important.

More generally, they conclude that what we need is more along the lines of a right to contest, or to demand reconsideration, than an actual explanation. What someone who has been adversely affected by an opaque agency decision:

“Needs is an ability to ask the experts at the agency to look again, not the ability to comb through information to find out herself [what went wrong]. Especially given how overwhelmingly costly such transparency would be, the negligible epistemic benefit for the vast majority of inquirers cuts hard against the idea that such benefits should be enshrined in anything like a right”

Without debating their larger claims either way, I think the superfluity of a right to explanation is based on a couple of points which interact strangely with the current U.S. legal system.  These are barriers to effectuating an explanation without a right, and in that sense they raise the cost of not having a right to an explanation.

In defense of superfluity, Karlan and Kugelberg write:

“What is needed is often not a complicated technical explanation of the algorithm itself, but an improvement in the second-order knowledge or algorithmic literacy of the recipients of decisions (Karlan & Allen, 2022). Second-order knowledge includes many things that can be known about the algorithm without knowing anything about its underlying architecture. Complete black boxes, for instance, can still be analyzed for a mapping between their inputs and outputs, and algorithmic audits can find vulnerabilities and biases in algorithms whose architecture is a trade secret. Combine this with a general understanding of how algorithms classify data, a hallmark of the nascent algorithmic literacy movement (see Oeldorf-Hirsch & Neubaum, 2025), and often it will be unclear what else will be needed to put those on the end of an algorithmic decision in a position to contest that decision”

In other words, if our concern is fairness, then algorithmic audits by people (“or their lawyers”) ought to be enough to root out the fact of unfairness.  They cite ProPublica’s well-known investigation of the COMPAS algorithm. This is fine, but in the U.S. at least, I think it runs up against judicial limits on class action litigation. Those who are subject to adverse decisions often suffer a harm that is too small to litigate individually.  The cost, both in terms of resources and time, is simply too high to be worth it.  For example, if I am denied a car loan because of my credit score, or a landlord refuses to rent to me because of some sort of abstruse algorithm, the total harm to me is far less than it would cost me to litigate any sort of compensation.  A similar scenario would apply to many benefits. So it’s irrational to go to court unless a group of individuals can go together as a group, as a class of folks with a “common” attribute.

Aside from how long it takes, one problem with class litigation is that the current Supreme Court is very hostile to class certification.  I’ll list a couple of examples.  First, TransUnion v. Ramirez (2021; my thoughts on it and what it means for standing more generally: part 1, part 2). In that case, TransUnion wrongfully asserted that thousands of people were on a terrorism watch list because of bad data-matching practices.  The Supreme Court ruled that only those whose erroneous match had been given to a third party had standing to sue, nevermind that the credit agency’s business model was to sell this information.

Second, the court reads “common” extremely narrowly. Consider, for example, Wal-Mart v. Dukes (2011), which concerned whether 1.5 million female Wal-Mart employees could be certified as a class in order to pursue an employment discrimination claim against the company.  In essence, their claim was the “local managers’ discretion over pay and promotions is exercised disproportionately in favor or men, leading to an unlawful disparate impact on female employees” (344).  In other words, Wal-Mart fails to rein in its local managers, creating a problem which is legible on the structural level.  Writing for the majority, Justice Scalia argues that class members’ “claims must depend upon a common contention” which “must be of such a nature that it is capable of classwide resolution – which means that the determination of its truth or falsity will resolve an issue that is central to the validity of each one of the claims in one stroke” (350).  Plaintiffs produced evidence that “there are statistically significant disparities between men and women at Wal-Mart” which “can be explained only by gender discrimination,” and that “Wal-Mart promotes a lower percentage of women than its competitors” (356).  Scalia wasn’t buying: for him, “common” means an attribute that is predicable of every member of the class, as “a regional pay disparity, for example, may be attributable to only a small set of Wal-Mart stores, and cannot by itself establish the uniform, store-by-store disparity upon which the plaintiff’s theory of commonality depends” (357).

Finally, it is worth noting that many, many potential class plaintiffs will not get their day in court at all.  The Court has been very aggressive in interpreting the century-old Federal Arbitration Act to mean that agreements to arbitration are binding no matter what the complaint is, and within that, it has been particularly hostile to class arbitration.  Since virtually all consumer transactions include mandatory arbitration clauses, access to the courts or to class-based arbitration will be closed to most plaintiffs.

All of which is to make a point about the intersection between moral theory and the current judicial system.  I worry that the argument that lawyers can root out the fact of unfairness won’t work – because there will be no lawyers or plaintiffs incentivized and empowered through discovery to do so.

But there’s a second problem. Even if the fact of fairness is rooted out, and plaintiffs can get to court with class certification (or even, hypothetically, individual standing), it’s not clear to me that this will be enough. Karlan and Kugelberg write:

“It seems to us that if it is determined that a system generally produces unfair or biased outputs, then this gives those who have had their cases tried by the system strong grounds for complaint. The arguments they can use in their informed self-advocacy are straightforward: ‘I belong to group X. The system has proven to be biased against members of group X. I therefore demand a new and fair hearing of my case, using a different decision procedure.”

But here again, the legal system is a problem.  The plaintiff in this case has basically raised a disparate impact claim.  The current Supreme Court likes those even less than it likes class actions.  To prove discrimination these days often requires a smoking gun, where some bureaucrat loudly announces their intent to discriminate.  That’s not going to happen with algorithms.  A robust legal literature dating to about 2016 (see Barocas and Selbst’s “Big Data’s Disparate Impact” and Margaret Hu’s “Algorithmic Jim Crow” for examples) established that algorithmic systems might well produce disparate impact problems because of features like proxy variables (the system might be barred from using race, but it might use zip code which in most U.S. jurisdictions is a close proxy).  But the current SCOTUS has all but green-lit actual Jim Crow voting laws in Louisiana v. Callais, and in any case it tends to treat disparate impact claims with a skeptical eye. Federal agencies like the DOJ and EEOC under Trump are close to refusing to look at disparate impact claims at all.

Where does this leave us? I am fully persuaded by the climate change example and the need to be able to incorporate good science into high-level policy. I can’t decide whether I am persuaded that the ability to contest algorithmic decisions requires an antecedent explanation.  I can see both sides of that argument; I am more persuaded of the stark decline in the rule of law that losing explanations might entail (that’s obviously too big for here, but this paper (preprint) will give you some idea of where I currently am on that).  But the ways that you could contest a decision without an explanation are difficult in the U.S. In that sense, it seems to me like a good example of the limits of ideal philosophy.