Code you’ve never read

The four horsemen of not invented here

Resistance to new programming tools is one of the oldest continuous traditions in the field, and after seventy years it has settled into a stable script. Whatever the tool—the compiler, the shared library, the garbage collector, the package manager, now the model—the objections arrive in the same four shapes.

It makes you lazy. Lean on it and the muscle wastes. You don’t understand what’s underneath. The abstraction hides the machine, and one day the machine will surprise you. It’s unreliable. It produces code that looks right and isn’t. Real engineers don’t need it.

Each is, in its moment, partly true—which is what makes the script so durable. It is a real cost, mistaken for a permanent one, in an argument had over and over about tools nobody would now give up.

“Efficient programming cannot be automated”

Before compilers there was hand-coding, and the people who did it regarded their work, in John Backus’s own words, as “a complex, creative art that required human inventiveness.” Early attempts at “automatic programming” had earned their skepticism fairly: they ran the machine five or ten times slower than a person would, which had “convinced programmers that efficient programming was something that could not be automated.”

So when Backus’s team built FORTRAN in the mid-1950s, the burden of proof was entirely theirs: the language would be used only if it produced programs “almost as efficient as hand coded ones and do so on virtually every job.” Audiences in 1954 and 1955 were “mostly skeptical,” having heard too many glowing descriptions of systems that turned out to be clumsy.

“It is difficult for a person who did not live through this period to appreciate the strength of the skepticism about automatic programming.” John Backus, recalling 1954

The skeptics lost, and quickly. Within a couple of years most of the code running on IBM’s machines came out of the compiler. When Grace Hopper later proposed a language built from English words, the story she told was that she was informed it could not be done because computers did not understand English—and that it took three years to be taken seriously. The objection was not that her idea was hard, but that wanting the machine to meet people halfway was a category error.

“Using someone else’s routine is cheating”

The idea that a programmer might pick up working code someone else had written is now so ordinary it is invisible. It had to be invented, and then trusted, and the trusting was the hard part. The first documented library of reusable code appears in 1951, in a book by Maurice Wilkes, David Wheeler, and Stanley Gill describing Cambridge’s EDSAC machine: a catalogue of subroutines, punched onto paper tape, that a programmer could thread into a program without writing them fresh—the ancestor of every import statement since, carrying the same proposition that unsettles people today, that you might rely on lines you did not write and had not checked.

By 1968 the ambition was enormous. At the NATO Software Engineering Conference, Doug McIlroy’s talk “Mass Produced Software Components” argued that software should be built the way mechanical engineering builds things—from families of standard, catalogued parts a builder selects rather than forges—changing the first question of any project from what shall we build? to what mechanism shall we use? It reads now as a plain description of the package ecosystems every developer lives inside. In 1968 it read as a provocation.

Engineering Darwinism, on a loop

Once a tool clears the bar of “does it even work,” the argument retreats to firmer ground: not that it fails, but that it weakens the people who use it. This is the crutch argument, and it endures because it can never quite be disproven.

Its cleanest instance is the war over the humble goto. In March 1968, Edsger Dijkstra published a letter in Communications of the ACM titled “Go To Statement Considered Harmful”—the statement was “too primitive,” he wrote, “too much an invitation to make a mess of one’s program.” What followed was a decade of trench warfare. The programmer P. J. Plauger remembered converts waving the new gospel of structured programming under the noses of “the unreconstructed assembly-language programmers who kept trotting forth twisty bits of logic and saying, I betcha can’t structure this.”

The pattern recurs everywhere since. Automatic memory management was for people who could not be trusted to manage their own. Editors that finished your function names were making programmers soft. Syntax highlighting was decoration; version control, ceremony. In 1983 an essay called “Real Programmers Don’t Use Pascal” made the whole disposition into a joke it was in on. Every one of those tools is now part of the floor a programmer stands on. That is what winning looks like in this field: not vindication, but invisibility.

Code no one has read

Here is what the crutch argument gets right. Modern software runs on an immense foundation of code that nobody involved has read. Install a mid-sized project and a machine downloads hundreds of packages written by strangers, most of them depending on further strangers, and wires them into the thing that will handle real users’ data. The trust is total and almost entirely unexamined.

The clearest look anyone got at that foundation came in March 2016, when a developer named Azer Koçulu, in a dispute over a package name, unpublished 273 of his modules from the npm registry. One was left-pad: eleven lines of code that added spaces to the front of a string. Within hours, builds were failing across the industry—Babel, React, and everything downstream—at hundreds of failures per minute. Eleven lines, written by one person, quietly load-bearing under a meaningful slice of the web.

“Have we forgotten how to program?” David Haney, in the days after left-pad

It is a fair question, and the same one the FORTRAN skeptics asked. Yes, the field has forgotten—deliberately, the way it forgot how to hand-assemble loops. Every layer of abstraction is a decision to stop holding something in your head so you can hold something larger. “Code you did not write and cannot see” is not a property of AI; it is the water the discipline has swum in since a subroutine went onto paper tape at Cambridge.

The myth of eyeballs

If the worry is trusting code strangers wrote, there is a comforting answer: open source. Millions of eyes on the code, the reasoning goes, so anything dangerous gets caught. Eric Raymond gave the idea its slogan—“given enough eyeballs, all bugs are shallow”—and it hardened into a half-belief that open, widely used code is, by virtue of being open and widely used, safe.

But openness is permission to look, not proof anyone did. In 2018, the maintainer of a popular JavaScript package called event-stream, Dominic Tarr, handed publishing rights to a stranger who had volunteered to help—an entirely normal thing to do, he noted, since sharing commit access had been common “since the early days of node/npm.” The stranger used it to slip in code that tried to steal cryptocurrency. “If I had realized they had a malicious intent I wouldn’t have,” Tarr wrote, “but at the time it looked like someone who was actually trying to help me.”

The same story, at its most patient, is xz-utils. From around 2021, someone operating as “Jia Tan,” backed by sock-puppet accounts applying pressure, spent years earning the trust of the sole unpaid maintainer of a compression library buried inside nearly every Linux system. By 2024 they had co-maintainer access and used it to plant a backdoor of the maximum severity a vulnerability can carry. It was found by luck: a Microsoft engineer, chasing half a second of unexplained delay in an unrelated benchmark, pulled the thread until the whole thing unraveled. Had he been slightly less curious, the backdoor would have shipped to the world.

The many eyes were not there. Critical infrastructure turned out to rest, in more than one place, on a single exhausted volunteer and the good faith of people nobody had verified. This is not an argument against open source, one of the great achievements of the field—only against the myth that “lots of people use it” is the same as “someone is watching it.” Trust in code has always been mostly social, and only occasionally verified.

When software was a giveaway

The very idea of software as a serious thing—built by professionals, sold, defended—is itself a recent settlement, and every step toward it was contested. In the beginning software was a giveaway: hardware makers bundled it in free, because the machine was the product and the programs an accessory to move iron. That changed in 1969, when IBM, under antitrust pressure, unbundled its software from its hardware and began charging for it separately. Software turned, as the Computer History Museum puts it, “from a giveaway to a competitive commercial product,” and an entire industry was born.

A year earlier, at Garmisch, the field had named its anxiety the software crisis: systems had outgrown the craft methods used to build them—OS/360 was said to have cost more than fifty million dollars a year and five thousand person-years—and the cure was to stop treating programming as an art and start treating it as engineering. And when a movement later arose to make software free and open again, it was met as menace: in 2001, Microsoft’s Steve Ballmer called Linux “a cancer that attaches itself in an intellectual property sense to everything it touches.” The thing later generations would invoke as the definition of trustworthy code was, within living memory, the disease. Today’s normal is reliably yesterday’s scandal.

What’s actually new

None of this proves the skeptics wrong about AI, and it would be its own laziness to answer “this has all happened before” and stop thinking. Some things genuinely are different: a compiler is deterministic where a model is not; a library has an author, a version, a place to file the bug, where generated code often has none of that lineage; and the volume of code a small team can now emit outruns its capacity to review it.

But these are complaints about reliability and about not understanding what’s underneath—the second and third lines of the old script, argued with unusual force. A powerful, unreliable tool is exactly the situation the field has been in before, and the response that worked was never refusal. It was learning where to trust the tool, building the checks that made the trust safe, and moving the craft up to the parts the tool could not do.