Open a terminal and type a command you’ve never run before. You can read its source, trace its output, pipe it into another command, undo whatever it did. Now try to find out why your loan application was denied, why your insurance premium went up, why a hiring algorithm ranked you below a stranger. You can’t. Not because the reasoning is too complex to explain — because no one who built the system intended for you to see it. Put the two side by side and a question opens up that has nothing to do with computers specifically: what obligation does a system owe the people it governs, and why do some systems honor it while others treat opacity as a feature?
I. A Requirement Hiding in Plain Sight
An earlier post on the Unix shell praised it as the most durable interface in computing history, and it is durable for a specific reason: every command is text, every script is inspectable, every pipe connects two transparent processes rather than two black boxes. That praise wasn’t really about elegance. It was about a property the shell happens to have and most modern systems don’t — the property of being readable by the people who use it. Once you notice that property, you start seeing its absence everywhere: in the algorithm that sets your credit score, in the neural network that flags your medical scan, in the filing cabinet of unwritten rules that decides whether your appeal gets heard. These look like unrelated problems — finance, medicine, bureaucracy — but they share a single defect. Call it the audit imperative: a system that governs people must be readable by those people, and a system that cannot be read will eventually serve interests other than theirs.
II. What Readable Actually Means
Readable isn’t a vibe, it’s four conditions, and a system either satisfies them or it doesn’t. The rules have to be published — not just their existence, but their full content, in a form a non-expert can understand, which rules out “the algorithm is proprietary” as an acceptable answer to “why was I denied.” The application of the rules has to be traceable — for any specific decision, you can reconstruct the inputs, the process, and the reasoning that produced it, not just receive the output. The system has to be modifiable — when the rules produce bad outcomes there’s a mechanism for changing them that doesn’t require the permission of whoever benefits from the current ones. And exit has to be real — if the system can’t be made readable, the people subject to it can leave without catastrophic loss. Miss any one of these and you don’t have a slightly-worse version of an auditable system. You have an unaccountable one, because the four conditions aren’t a checklist of nice-to-haves, they’re the mechanism by which power gets checked at all.
III. The Original Audit Mechanism
The oldest working example predates computing by five centuries. An earlier post on accounting traced how double-entry bookkeeping forces every transaction into two independent records that must agree — debit here, credit there, and if they don’t match, something is wrong and the discrepancy cannot hide. That’s the audit imperative in its purest mechanical form: not a promise of honesty but a structure that makes dishonesty visible by design. Luca Pacioli didn’t invent double-entry to make merchants virtuous. He invented it to make their books legible to anyone who could read a ledger — a partner, an heir, a tax collector, the merchant’s own future self trying to remember what happened. The system didn’t need the merchant to be trustworthy. It needed the merchant’s records to be checkable, which is a categorically weaker and far more durable requirement.
IV. When the Ledger Has No Second Column
Most modern governance systems fail this test in the same specific way: they have a first column and no second one. A credit-scoring algorithm, a bail-risk assessment, an insurance-pricing model — each produces a number, and the number is treated as the record. There’s no independent entry it has to agree with, no discrepancy that can surface, because there’s nothing to compare it against except itself. An earlier post on neuro-symbolic AI made the architectural version of this argument: symbolic reasoning is traceable step by step, the way a ledger entry can be traced to a transaction, while a neural network’s decision is a weighted sum across millions of parameters that no one, including the people who trained it, can walk back into a reason. The problem isn’t that these systems are wrong more often than humans — in narrow domains they’re frequently wrong less often. The problem is that when they are wrong, the error has no address. Nobody can point to the entry that doesn’t balance, because there was never a second column to check it against.
V. The Symbolic Precedent
An earlier post on Nefasto described a 1989 Prolog program built to satirize the opacity of academic discourse — pompous, evasive language generated by rules the program’s author could point to and name. The joke worked because Prolog is a system where every inference is traceable to a premise: ask it why it concluded something and it can show you the chain. The target of the satire, academic jargon designed to sound authoritative without committing to a checkable claim, was the bureaucratic black box wearing a tweed jacket. Thirty-seven years later the shape of the problem hasn’t changed, only its instrument — a language model can now generate the same evasive plausibility at a scale no human committee ever could, and unlike the Prolog program, it can’t show you the chain, because there isn’t one to show.
VI. The Fuse That Went Missing
An earlier post on medical advice and LLMs named the specific harm that shows up when a system this opaque gets inserted into a decision that used to require a licensed, accountable human: the fuse between suggestion and consequence disappears. A doctor who gets something wrong can be asked to explain the reasoning, can lose a license, can be sued, can be wrong in a way that leaves a record and a name attached to it. A model that gets something wrong produced a plausible-sounding sentence, and the chain of accountability terminates in a terms-of-service disclaimer. This is the audit imperative’s fourth condition failing quietly: exit isn’t the problem here, traceability is — there’s no mechanism by which the person harmed can reconstruct why the harm happened, only a confident paragraph and no one obligated to stand behind it.
VII. Why “Proprietary” Isn’t an Answer
The standard defense against all of this is that the algorithm is proprietary, a trade secret, competitively sensitive — and there’s a real distinction hiding inside that defense worth preserving rather than dismissing. A company keeping its recipe secret from competitors is not the same claim as a company keeping its decision logic secret from the person the decision was made about. The first is ordinary competition. The second is a system’s subjects being denied the one thing that would let them check whether the system is treating them fairly, and dressing that denial in the language of intellectual property doesn’t change what it is. A useful test: would the explanation, if published, help a competitor copy the business, or would it only help the affected person understand their own case? Credit-scoring weights are the first kind of secret sometimes and the second kind always — and a rule that can’t survive being explained to the person it was applied to was probably never a legitimate rule to begin with.
VIII. The General Shape
None of this is really about algorithms. Algorithms are just the newest and fastest-scaling instance of an old failure mode — opaque bureaucracy did this first, rules that technically exist but are never published, decisions made but never explained, appeals that are theoretically possible but practically inaccessible to anyone without a lawyer on retainer. The mechanism is identical whether the black box is silicon or paperwork: opacity protects the decision-maker from the discipline of being checked, and a system does not need malicious intent to drift toward serving its operators over its subjects — it only needs the absence of a mechanism that would catch the drift. The shell stayed honest for fifty years not because its designers were unusually virtuous but because every command remained visible to the people who depended on it. Double-entry bookkeeping caught fraud for five centuries not because merchants became more honest but because the ledger had two columns instead of one. The audit imperative isn’t a plea for good behavior. It’s the observation that good behavior is not the thing to design for — checkability is, because checkability is the only property that survives contact with an operator who doesn’t feel like behaving.
