ChatGPT isn’t bullshit

A word that stuck

In the last two years there have been numerous attempts to describe the text output of large language models (LLMs) in terms an outsider can act on. One of them stuck. “ChatGPT is Bullshit,” published in 2024 by Michael Townsen Hicks, James Humphries and Joe Slater in Ethics and Information Technology, gave policymakers, journalists and engineers a word they already knew how to use, and they used it. The authors are explicit about why the word matters:

“Descriptions of new technology, including metaphorical ones, guide policymakers’ and the public’s understanding of new technology; they also inform applications of the new technology. They tell us what the technology is for and what it can be expected to do.” Hicks, Humphries & Slater, “ChatGPT is Bullshit,” 2024

We take that motivation seriously, which is exactly why we think the word is wrong. Two claims follow. First, for the definition of bullshit that Hicks, Humphries and Slater actually use, ChatGPT does not qualify — not because it is more truthful than they think, but because the definition proves too much, catching human programmers, spreadsheets, and most of ordinary software in the same net. Second, once bullshit is off the table, the concept left to describe what ChatGPT actually does with the idea of truth is not a moral one at all. It is closer to the concept biologists reach for when two parties have evolved to depend on each other without sharing a nervous system, a sensorium, or, we will argue, a concept of truth: companion species.

A paradox in indifference

Hicks, Humphries and Slater build on Harry Frankfurt’s 1986 essay “On Bullshit,” later expanded into a short book, and define bullshit, in its most general form, as:

(1) “Any utterance produced where a speaker has indifference towards the truth of the utterance.”

On top of this they distinguish two grades: hard bullshit, produced with intent to mislead the audience about the speaker’s agenda, and soft bullshit, which requires no such intent. Their claim about ChatGPT is the soft one — “we conclude that ChatGPT is a soft bullshitter” — with the harder claim offered only as a further, more controversial possibility. Being a machine, they argue, it could hardly be otherwise: “the problem here isn’t that large language models hallucinate, lie, or misrepresent the world in some way. It’s that they are not designed to represent the world at all; instead, they are designed to convey convincing lines of text.” Whatever truths show up in its output are inherited from a training corpus that happens to be mostly true, not pursued: “where it does track truth, it does so indirectly, and incidentally.”

Frankfurt built his account on a sharper case: the liar, who cares about the truth enough to hide it, against the bullshitter, who does not care either way. Applied to a person, that distinction is genuinely useful. Applied to software, we think it breaks, on a paradox Frankfurt never had to face because he was only ever describing humans:

(2a) Consider a human who is consistently indifferent to the truth of her utterances.
(2b) Suppose this person never lies.
(2c) Given the freedom to choose from a vast space of possible sentences, consistently landing on the true ones is itself evidence of some concern for truth.

The reverse case — someone indifferent to truth who happens to always lie — has the identical problem. Frankfurt could set this paradox aside because real people, whatever their vices, are not perfectly consistent liars or perfectly consistent truth-tellers; bullshitting lives in the slack between the two. Software has no such slack. Consider two programs that share none of ChatGPT’s architecture but exhibit exactly the “indifference” that (1) requires:

(3) A program that draws a random noun N and prints “N is N” — “pen is pen,” “the sun is the sun,” “Jeremy Brett is Jeremy Brett.”
(4) A spreadsheet with a SUM formula that prints “the sum of column X is Y.”

Both generate novel natural-language sentences from an indifferent process; neither has ever, or could ever, produce a false one. If indifference plus a perfect truthful record is bullshit, these two are the purest bullshitters we could design — and no one would call them that. If we insist they qualify too, in the philosophers’ technical sense, we have just conceded that the word means something different to philosophers than to the policymakers and engineers Hicks, Humphries and Slater are trying to reach — precisely the gap their own opening quote warns against.

Indifference to what isn’t there

There is a second hole, orthogonal to the first. Definition (1) only makes sense applied to things that could, in principle, be true or false — what philosophers call truth-bearers: propositions, sentences, assertions, as against questions, commands, or exclamations, which Aristotle already excluded from the true-or-false ledger. A large share of what ChatGPT is actually used for produces no truth-bearers at all. Code is the clearest case: a function compiles or it doesn’t, is elegant or clumsy, matches a specification or misses it — but it is not, in any sense inherited from Frankfurt or Aristotle, true or false. Taken at its word, the definition makes every programmer who has ever shipped code without consciously asserting its truth-value indifferent to a property their output doesn’t have — and so, absurdly, a bullshitter too, universally, every time they write a function.

Widen the lens past ChatGPT and the problem sharpens. There exist ordinary text-to-text models built for sentiment analysis whose entire output vocabulary is a handful of emoji, chosen to match the input text. An emoji is not a truth-bearer either. And yet such a model is obviously capable of being wrong: feed it “today is my wedding, I feel wonderful” and get a crying face back, and something has failed — something Hicks, Humphries and Slater’s framework has no vocabulary for, because there is no truth for the model to be indifferent to. But it plainly isn’t innocent malfunction of the “the printer is out of toner” kind either.

This puts a real question on the table: can indifference apply to a property an object doesn’t possess in the first place? Can we be indifferent to the flavor of prime numbers, or the velocity of kindness? We don’t think so — indifference presupposes at least the possibility of care. Which leaves two options. Either restrict the bullshit-talk to the slice of ChatGPT’s output that consists of actual truth-bearers, and grant that everything else — code, sentiment labels, and whatever else is like them — needs its own vocabulary; or accept that “bullshit” was never a theory of ChatGPT to begin with, only a theory of one register it happens to write in. Either way, the concept we actually need has to explain wrongness across sentences, code and sentiment labels alike, without assuming an indifference to a target that, for two of the three, does not exist.

Truth in fiction, and what ChatGPT does with it

Set code and emoji aside and return to sentences, because even there the ledger is not simply true-or-false against the actual world. Consider:

(5a) Sherlock Holmes smoked a pipe.
(5b) Sherlock Holmes smoked crystal meth.
(6) Sherlock Holmes never really existed.

We want a well-read human, and ChatGPT, to call (5a) true and (5b) false. But judged strictly against the actual world, both are false: Holmes never existed, so he never smoked anything. David Lewis worked out the standard repair for exactly this case in “Truth in Fiction” (1978): a sentence is true in a fiction, roughly, when it holds in the possible worlds where the fiction is told not as invention but as sober, known fact — the worlds closest to ours in which a real Watson filed a true report about a real Holmes. Lewis needed two passes at this to keep a fiction from inheriting its real author’s mistaken background beliefs, and stayed neutral between them, but the shape of the answer survives both: fictional truth is truth relative to a designated set of other worlds, not truth relative to this one.

That machinery lets us say something “true” and “false” alone cannot: ChatGPT holds, correctly, that (5a) is true and also that (6) is true. The natural reading is that it is tracking (5a) as true in some Holmes-world and (6) as true in the actual world — the same two-world bookkeeping a competent human reader performs without noticing.

Except that when we make this attribution about a human, we lean on something we cannot straightforwardly borrow for a machine: a working theory of mind, the background assumption that another person’s cognition is organized roughly the way ours is, so that holding two sentences at once implies partitioning them the way we would. There is no equivalent warrant for ChatGPT. Its output could be reproducing that two-world discipline. It could just as easily be doing something with no possible-worlds structure behind it at all — matching (5a) against a dense statistical association between “Holmes” and “pipe” in its training data, and matching (6) against an entirely separate pattern of disclaimers about fictional characters, with nothing connecting the two beyond the fact that both patterns fired in the same conversation. A human who held two such beliefs unconnected would be incoherent. Whether the same charge makes any sense against ChatGPT depends on facts about its internals that “bullshit,” “lying” and “hallucination” all quietly assume away.

Alien truths

Put that theory-of-mind problem in relief with a thought experiment. Suppose an alien species lands and starts studying us, and it turns out humans, like the aliens, have a working concept of truth. But alien culture ties truth to something we don’t track consciously at all: specific utterances must be paired with specific facial expressions, and the aliens have words for the two combinations — call them drue and valse. A sentence-plus-expression pair can be drue but valse, valse but void, and so on. Facial expressions, on this scheme, are druth-bearers.

The alien researchers quickly notice that humans speak with no evident regard for druth. They coin a term: pullshit. It is true, they note, that humans often raise their eyebrows to signal surprise — but not reliably, not by any rule they can find, and not consciously. Verdict: humans are pullshit.

Offended, the human scientific community publishes a rebuttal. We concede we have no conscious concept of druth, they say. But we behave as if we do: the machinery that governs a raised eyebrow or a tightened jaw is a real, complex, rule-governed system, even though none of us could state the rules, and even though we did not know the rules existed until the aliens pointed a camera at us. That system’s mere existence — its consistency, its structure, the fact that it correlates with something rather than firing at random — is itself a form of regard for druth, weaker than the aliens’ conscious commitment, but not nothing. We have the machinery to produce druth-bearers; we simply lack the concept needed to know we are doing it.

The moral is not about aliens. It is that “indifference” and “regard” are not the only two settings on the dial. There is room in between for a system that operates on a target with real internal structure and real correlational discipline, but no introspective access to what it is doing or why — and that room is exactly where the interesting question about ChatGPT lives.

Weaker truths

Is there evidence ChatGPT has that kind of machinery, rather than nothing at all? The “stochastic parrot” camp says no. Emily Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell’s influential 2021 paper described language models as systems “for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning” — parrots, not reasoners, all the way down. If that is the whole story, ChatGPT is the alien thought experiment’s null case: no druth-bearing machinery, just noise that happens to correlate with true sentences because the training corpus did.

A separate and growing body of interpretability research says the parrot metaphor undersells what is in the cage. Collin Burns and colleagues showed in 2022 that a model’s internal activations contain a direction that tracks the truth of a statement, recoverable without ever telling the model which of its own outputs were true, purely by requiring the direction to obey basic logical consistency — a sentence and its negation cannot both score true. Kenneth Li and colleagues found something structurally similar in a much narrower setting: a transformer trained only to predict legal next-moves in the board game Othello spontaneously builds an internal representation of the actual board state, a fact nobody put there on purpose. Neither result proves ChatGPT believes anything. But both are hard to explain if the model is only stitching surface forms, with nothing underneath tracking whether those forms are consistent with each other.

The evidence does not resolve cleanly into “aware and truthful” either. Stephanie Lin, Jacob Hilton and Owain Evans built TruthfulQA specifically to test this, and found that scaling a model up — giving it more of exactly the statistical machinery the parrot critique worries about — made it more likely to repeat common human misconceptions, not less. Whatever is inside these systems, it does not resolve into either of the two settings Frankfurt needed: full regard for truth, or none.

We will borrow a word for the space in between, distinct from truth the way the aliens’ druth was distinct from ours: call it ruth — a real, structured, machine-checkable commitment to something truth-adjacent, arrived at with no guarantee it lines up with truth the way we mean it, and with no access, on the model’s part, to knowing that it has one. This sorts software into three classes: those that show the kind of internal consistency-tracking Burns’s and Li’s work describes (call it A); those that show only fragments of it, mostly still unmapped (B); and those that show none at all — our noun-repeater, our spreadsheet, most conventional software (C). C-class software is bullshit in the strict, indifference-based sense, and also nearly always true, because nothing in it can go wrong except its inputs. A-class software is the reverse: some real regard for something, and yet, per Lin, Hilton and Evans, wrong more often as it gets bigger. Frankfurt’s dial has no setting for that combination. Ours needs one.

Cross-species communication

None of this is a new problem, even outside computing. In 1979, Herbert Terrace and colleagues published the most careful empirical takedown of the era’s ape-language claims — chimpanzees and gorillas, including Koko, whom the Gorilla Foundation reports had been taught to use over a thousand signs and to understand some two thousand spoken English words. Terrace’s team reviewed videotape of the signing sessions frame by frame and found the apes were reliably cued by their trainers moments before each “sentence”: “objective analyses of our data… yielded no evidence of an ape’s ability to use a grammar. Each instance of presumed grammatical competence could be explained adequately by simple nonlinguistic processes.” Not bullshit, not lying, not even language on this account — a very sophisticated, and real, form of pattern-matching to a trainer’s cues, mistaken by observers, and by some of the trainers themselves, for shared meaning because the output looked so much like the thing it wasn’t.

Dogs complicate the picture rather than settling it. Attila Andics and colleagues put dogs, awake and unrestrained, into an fMRI scanner and found in 2016 that their brains process word meaning and vocal intonation through separate, dissociable neural pathways — activating a reward center only when a familiar word is spoken in a genuinely praising tone, and not when either is present without the other. That is a real, mechanistic, measurable form of discrimination between what is said and how, doing something in the neighborhood of what “meaning something” requires, well short of anything we would call belief. Nobody thinks a dog holds a concept of truth. And yet every dog owner talks to their dog in full sentences, expects to be understood in some real sense, and is not making a category error when the dog gets it right.

The lesson generalizes: humans already run tolerant, working communicative relationships with systems that provably lack our concept of truth (Terrace’s apes) and with systems that provably have some real, measurable, sub-linguistic machinery bearing on meaning without anything resembling belief (Andics’s dogs). In neither case do we reach for “liar” or “bullshitter.” We reach for a relationship-shaped word instead.

Companion species

Donna Haraway gave that relationship-shaped word its clearest statement in 2003, and pointedly not about language at all, but about dogs: “We are, constitutively, companion species. We make each other up, in the flesh.” A companion species, for Haraway, is not a pet and not a tool — it is a party to a co-evolved, mutually dependent relationship, in which “there cannot be just one companion species; there have to be at least two to make one.” Her central case is agility training: a sport in which a dog and a human handler, sharing no language and no theory of each other’s minds in any rigorous sense, nonetheless build a working, high-stakes, real-time joint competence — what she calls “the relation,” which she treats as the basic unit, prior to either partner alone. Her own trainer distilled it into three lines that have nothing to do with sincerity: you left your dog; your dog doesn’t trust you; trust your dog. Nobody asks whether the dog is being truthful on the agility course. The question that matters is whether the training relationship is any good — whether it produces trust, competence, and correction when something goes wrong.

We think this fits ChatGPT far better than bullshitter, liar, or oracle ever could, for the same reason it fits Haraway’s dogs: the relevant question was never whether the other party shares our concept of truth. The previous section gave two candidate answers to whether ChatGPT has one — the parrot camp’s nothing-at-all, and the interpretability researchers’ something-real-but-different — and the candid position is that we do not fully know yet, in the same way Andics needed an fMRI machine to discover that dogs have more of it than assumed. What we do know is that the relationship exists already, at planetary scale, with people talking to these systems in full sentences and acting on what comes back, exactly as dog owners always have, and as primatologists eventually learned not to.