Direct manipulation and the shape of AI interfaces

The metaphor we thought we had left

Before the graphical interface there was the command line, and under the command line was a metaphor: conversation. You issue an instruction in a language the machine parses. It performs operations you cannot see. It prints a result. To do the next thing you must know what to say, and then reconstruct, from the reply, what happened and what the world now looks like. The command line is enormously powerful and almost entirely expert-only, and the two facts have one cause: the state of the work lives out of sight, and must be carried in the user’s head.

A large language model in a chat window is that metaphor with a spectacular parser bolted on. The improvement is real — you no longer need exact syntax, and the machine tolerates the way people actually phrase things. But the shape is largely untouched. You describe an intention in language; the model does work you cannot inspect; a passage of text comes back; and to revise, you describe again. The artefact you care about — the itinerary, the budget, the contract clause, the bridge — is never on the table between you. It is re-narrated, in prose, on every turn. Thirty years were spent moving the object of work out of the user’s head and onto the screen. The chat quietly moves a great deal of it back.

That this can feel like progress rather than regression is the puzzle the rest of this piece is built to resolve — because in one important sense it is progress, and in another it is not, and the two senses are usually run together.

What direct manipulation named

The term is Ben Shneiderman’s. He coined it in 1982 and set it out the next year in Direct Manipulation: A Step Beyond Programming Languages, abstracting a pattern from the systems users were visibly falling for — full-screen display editors, VisiCalc, spatial data management, video games — into three properties:

• continuous representation of the objects and actions of interest;
• physical actions — pointing, dragging, pressing labelled buttons — in place of command-language syntax;
• rapid, incremental, reversible operations whose effect on the object of interest is immediately visible.

The attribution has a wrinkle worth keeping, because it tells you the idea was recognised as a pattern before it had a name. When Hutchins, Hollan and Norman formalised it in 1985 they cited Shneiderman “(1974, 1982, 1983),” folding in a 1974 paper on interactive polynomial graphics as a conceptual precursor. The most defensible reading is narrow: the concept surfaces in the earlier work, the phrase is anchored to 1982, and the canonical statement is 1983. A pattern people already loved, named after the fact — the usual way genuinely load-bearing ideas enter a field.

What Shneiderman gave was a description and a norm. The norm is the important part: interaction should move away from command languages toward visible objects, labelled actions, immediate results, speed and reversibility — because those properties produce interfaces that are, in his later formulation, comprehensible, predictable and controllable. He was not cataloguing a style. He was naming a direction.

The cognitive account, and what chat does to the gulfs

Two years later, Edwin Hutchins, James Hollan and Donald Norman asked the question Shneiderman’s list left open — why do some systems feel direct? — in Direct Manipulation Interfaces (1985). Their answer is still the sharpest tool for this argument, so it is worth stating precisely.

Directness, they said, is the felt absence of distance, and distance comes in two kinds. The gulf of execution is the gap between what you want and the actions the system makes available to get there. The gulf of evaluation is the gap between the system’s output and your understanding of it. They split execution distance further into semantic distance (do the offered operations map onto the concepts you think in?) and articulatory distance (does the form of the expression relate naturally to its meaning?). Dragging a file to the trash has almost no articulatory distance: the action is the meaning. Typing rm -rf ./old has a great deal. Beneath distance they placed direct engagement: working on the objects of the task themselves, in a model-world, until “the interface and the computer become invisible.”

Now hold an AI chat against that framework — carefully, because the true answer is not “it widens everything.” Chat genuinely lowers semantic distance: natural language can express almost any intention, which is its real gift. It also removes a great deal of low-level action specification, which narrows part of the gulf of execution. That much is a true advance.

But the distance does not vanish; it moves, and mostly to worse places. Subramonyam and colleagues, in Bridging the Gulf of Envisioning (CHI 2024), extend Hutchins–Hollan–Norman directly: LLMs may narrow the gulf of execution, but they widen the gulf of evaluation and open a new gulf of envisioning — the user must now anticipate what the system can do, formulate an effective instruction, and predict the kind of output it will produce, all before acting. The intention still has to be flattened into language; the result arrives as prose describing what the model believes it did, which must be read, trusted and reconciled against a state that was never shown. And none of it restores the model-world. There is no persistent object between you; there is a description of one, regenerated each turn, and you are back to holding the real state in your head.

The directness ratchet

Here is the first claim. Directness was a persistent selection pressure under which the graphical interface evolved — not the only one, but a real and under-acknowledged one. The advances that lasted tended to shorten distance and widen the model-world; moves that reopened distance without paying for it elsewhere tended to be corrected. Call it the directness ratchet. It is a tendency, not a law — a later section owes it a full reckoning — but read the canon in this light and the pull is hard to miss.

Sketchpad (Ivan Sutherland, 1963) is the clearest prehistory. The user “sketches directly on a computer display” with a light pen; crucially, the sketch was not a picture of an object but a live, constraint-bound, editable object — alter a drawing and the geometry re-satisfies its constraints. Zero articulatory distance, and a model-world, in 1963.

Engelbart’s NLS (1968) pressed on the model-world from the other side. His programme was augmentation — raising the human capacity to “comprehend and solve complex problems” — and NLS was an instrument to compose, study and modify complex information structures from the inside, with the mouse, linked views and screen selection. The point was to work inside a structure rather than submit instructions to it from outside.

Smalltalk and the Star. Kay and Goldberg’s Personal Dynamic Media (1977) reframed the machine as a medium for the “communication and manipulation of knowledge” — not an appliance you operate but a material you work in. The Xerox Star (1981) turned that into a shipping doctrine whose stated principles were to reduce what the user had to type and to remember. Reducing typing and remembering is, precisely, closing both gulfs as policy.

The turtle (Papert, Mindstorms, 1980) is the odd entry and the most important for the AI argument. Papert’s turtle was an “object-to-think-with”; his microworlds were “incubators for knowledge.” A formal system becomes graspable when you can manipulate it through bodily and visual intuition. Directness, for Papert, was epistemic — it changed what you could understand, not only what you could do quickly.

VisiCalc (1979) took the ratchet mass-market. Change a cell and the whole model recomputes in front of you. It is still, by a wide margin, the most-used direct-manipulation tool on earth — and it won the office not by being conversational but by being manipulable: the numbers are the object, and you act on them.

The clearest modern statement is not a system but a talk. Bret Victor’s Inventing on Principle (2012) reduces the tradition to one demand — creators need an immediate connection to what they create — and every demo is the same manoeuvre: drive the latency between intention, representation and consequence to zero, so thought and artefact stay coupled. In Learnable Programming he turned it into criteria — follow the flow, see the state, create by reacting, create by abstracting — and in Explorable Explanations he extended it from making to reading, arguing text should become “an environment to think in.” This is the ratchet restated as a principle of creative work, and it is exactly the thing a chat turn breaks by inserting a round-trip of prose between the maker and the made.

A way to measure it

If directness is a direction rather than a style, it should be measurable, at least ordinally. The literature yields five questions to ask of any interface — three from Shneiderman, two from Hutchins, Hollan and Norman. This is the second claim: that AI interfaces can and should be scored on the same axis the graphical interface was climbing.

• 1. Continuous representation. Is the object of work persistently present, or reconstructed from scratch each turn?
• 2. Articulatory closeness. Do you act on the object, or describe an action in a language the object must be translated out of?
• 3. Incremental reversibility. Can you nudge, compare and undo in small steps, or must you re-issue a whole request?
• 4. Faithful evaluation. Does the shown state equal the true state, updated at once — or is it the system’s prose about its state?
• 5. Shared object-state. Is there one model-world that both you and the system act on, or does the artefact live only in the exchange between you?

Scored qualitatively, the axis sorts familiar things in a way that matches intuition — and locates the chat exactly where the cognitive account predicts.

The rubric is falsifiable in the ordinary way: if raising an interface’s score left task performance unchanged, the axis would be idle. The one careful study that did exactly this found the opposite. DirectGPT (Masson, Malacria, Casiez and Vogel, CHI 2024) wrapped Shneiderman’s properties around a language model — continuous representation of the generated objects, reusable commands, manipulable outputs, undo — and measured it against a conversational baseline on text-editing tasks. Holding the model constant, participants used 51.8% fewer prompts, wrote 72.3% shorter ones, finished 49.9% faster, and scored 24.3% higher on task completion. The model was not the bottleneck. The interface was.

The strongest objection, and the old fight

This argument has a serious opponent, and it is only worth making if it survives him. Jakob Nielsen, in “AI: First New UI Paradigm in 60 Years” (2023), divides computing into three eras: batch processing, command-based interaction, and now intent-based outcome specification. On his account conversational AI is emphatically not a return to the command line — the command line makes you specify how, while AI lets you specify what, which is a genuine break. His stock example is fair and forceful: describing a magazine cover to an image generator collapses what would have taken hundreds of Photoshop operations into one sentence.

He is right, and it sharpens rather than sinks the claim. “How versus what” and “directness” are different axes, and the mistake is to read a gain on one as a gain on the other. Intent-based specification lowers semantic distance — you can finally say the outcome you mean. It says nothing about whether the outcome then sits in front of you as a manipulable object or arrives as prose you must re-describe to change. The regression named here is not “chat makes you specify steps again” — it doesn’t. It is that chat, having won the how-to-what argument, threw away the model-world in the process, as if the two came as a set. They do not. And Nielsen half-agrees: in the same essay he calls today’s chat tools deeply flawed, notes they spawned the stopgap job of “prompt engineer,” and predicts the durable form will be hybrid.

The disagreement is itself a rerun. In the 1990s Pattie Maes argued that the direct-manipulation metaphor “requires the user to initiate all tasks explicitly and to monitor all events,” and would not scale — we would need agents to delegate to. Shneiderman defended visible, predictable control. Their 1997 debate is remembered as agents-versus-manipulation, but the half both conceded is the durable part: Maes said complement, not replace. Eric Horvitz made it precise a year later in Principles of Mixed-Initiative User Interfaces (1999), calling the goal an “elegant coupling” of automation with direct manipulation. The LLM re-fought that settled fight and, for a few years, took the losing corner — all delegation, no manipulable surface. The correction is now visibly under way.

A worked example: designing a bridge in the chat

The demonstrator below builds the top row of that table for a piece of genuinely hard creative work: an engineer and a model co-designing a 24 metre pedestrian footbridge. It is drawn as a drafting sheet rather than a chat app, on purpose — the visual language should belong to the work, not to the medium. Every force, deflection and collapse on the drawing is computed by an actual finite-element solver running in the page.

Three commitments carry it, and each answers a specific gulf named above:

• The widget’s state is the conversation’s state (test 5). There is one bridge. Every instrument reads and writes it, so when the crowd-scrubber discovers that the governing load case is a crowd at midspan rather than a uniform load, the member-sizing table sizes steel against that case, with no restatement. Nothing is re-described between turns because nothing was ever only prose.
• Manipulation is a speech act (tests 1 and 2). Dragging the depth handle is not a setting change; it is the sentence “make it deeper,” expressed with zero articulatory distance, and the model answers in kind: “Depth 2.6 m: chord force drops 600 → 415 kN, utilisation 0.78, deflection 36 mm — both checks pass.” The gulf of execution closes because the action and the intention are the same object.
• The model reacts to computed facts, not to itself (test 4, and the antidote to the gulf of evaluation). When the diagonal is cut, the truss does not merely “look unstable”: the stiffness matrix goes singular, and the collapse animation is that matrix’s actual null-space mode. The model’s prose is filled from solver output, so the sentence on the sheet cannot disagree with the drawing. There is nothing to audit, because the representation is ground truth rather than a rendering of the model’s claims about it.

What the language model does here is not diminished; it is aimed. It proposes the first truss, names the failure in words, suggests the counter-diagonals, argues for keeping the roller bearing — the things language is best at: framing, naming, proposing. Meanwhile the human keeps their hands on the object and a real engine keeps everyone grounded. Language proposes; manipulation disposes. That division is Horvitz’s coupling, forty years after he named it, finally cheap enough to build.

When directness is the wrong tool

A framework that only flatters its own thesis is not worth much, and the direct-manipulation tradition contains its own sharpest critique. Hutchins, Hollan and Norman set out in 1985 to give a cognitive account of the disadvantages as well as the advantages; directness was never claimed to be universal. It struggles with repetition, where a script beats doing the thing by hand a hundred times; with sets and variables, where “every file modified last Tuesday” is a description no amount of pointing expresses cleanly; and with precision and scale, where mimetic action becomes a burden. Their deepest warning is subtler: if designers only make the familiar easier to manipulate, they forfeit computing’s more radical promise, which is to invent new ways to think. Directness pursued shallowly is just skeuomorphism.

Each exception is exactly where language, and now the model, earns its place: the underspecified goal (“something like this, but calmer”), discovery before you know the object well enough to point at it, quantified and conditional operations over sets, and breadth across domains where no manipulable surface exists or would repay building. Nielsen’s magazine-cover example is precisely this territory, and it is a real win, not a concession made for form’s sake.

There is also plain evidence that less-direct interfaces persist because they solve problems directness does not. ChatOps — running deployments and inspecting results inside a team chat — endures because conversation can unify social coordination and operational control in a way a GUI alone often cannot. And directness has hard modality limits: pressed in the 1997 debate on blind users and speech, Shneiderman conceded that direct manipulation leans heavily on visual representation. “Direct” is not synonymous with “a visual GUI,” and treating it that way excludes people.

The deepest objection is historiographic. The ratchet is a teleological story, and teleology is how histories of technology most often go wrong. Haigh and Ceruzzi’s A New History of Modern Computing (2021) tells the field not as a march toward an optimal interface but as a sequence of communities remaking the computer for their own ends. So the careful form of the claim drops a notch in strength and gains in defensibility: directness is one selection criterion among several — control, learnability, scalability, accessibility, coordination, abstraction level, delegation burden — and GUI history repeatedly traded it against automation and abstraction rather than simply maximising it. Modern chat is therefore not a pure regression. The most defensible reading of where things are heading is hybridisation, not restoration — which is also what Nielsen, Maes and Horvitz each predicted from their different corners.

The correction, already under way

The claim that the field is turning back toward manipulable representations is not a hope; it is visible in the 2024–25 literature, and it clusters tightly around the rubric’s tests. The sharpest instance is AI-Instruments (Riche et al., CHI 2025), which names the exact problem — chat as “verbose linear-sequential texts” that make backing up and changing direction hard — and reifies prompts into persistent graphical instruments you can inspect, vary and reuse. Exploring Mobile Touch Interaction with LLMs (Zindulka et al., CHI 2025) turns pinch-and-spread into a control vocabulary for a model, acting on the text directly instead of breaking out into chat. LAVE (Wang et al., IUI 2024) lets users refine an agent’s video edits on the timeline rather than accept a conversational plan wholesale. And the whole tendency now has a venue: the MIND workshop at IUI 2025 — “Blending Agents and Direct Manipulation for Harnessing LLMs” — is the old Shneiderman–Maes question convened as a research programme.

The most consequential fork runs through the term “generative UI,” which already carries two incompatible meanings. In the runtime, end-user sense — the Nielsen Norman Group’s definition (Moran & Gibbons, 2024) — the AI generates the delivered interface itself, on demand, per query. Google’s Generative UI is the strongest instance: a model that generates “not only content, but the entire user experience,” preferred to plain markdown in 82.8% of raters’ pairwise comparisons. That is one path: a fresh, unpredictable interface every session — potentially low on continuous representation and reversibility, a new command line reinvented each turn.

The other path keeps the generated interface inspectable and manipulable. Generative and Malleable User Interfaces (Cao, Jiang & Xia, CHI 2025) grounds generation in a task-driven data model the user keeps editing, through language and direct manipulation. This is “generative direct manipulation,” and it is a different bet from “the model invents a new interface every time.” Which reading wins the word will matter more than it looks. Two further currents run the same way: malleable software (Ink & Switch, 2025) argues the object of manipulation should extend to the tool itself, and the tools-for-thought line (Matuschak & Nielsen) carries Papert’s and Engelbart’s epistemic strand forward. Even Shneiderman’s own recent human-centred AI programme fits: systems with both high human control and high computer automation. The through-line from a polynomial-graphics paper in 1974 to human-centred AI in 2020 is one person holding a position for forty-six years while the field arrives, twice, at where he was standing.

The stopgap consensus

It is worth being clear how much of the field now treats prompt-only interaction as temporary rather than settled. Meredith Ringel Morris titled a 2024 Communications of the ACM piece Prompting Considered Harmful, arguing plainly that prompting is a poor interface and should be phased out. Zamfirescu-Pereira and colleagues, in Why Johnny Can’t Prompt (CHI 2023), showed empirically that non-experts prompt opportunistically and brittly — the apparent naturalness is misleading. And the burden the chat pushes downstream, onto evaluation, is now measured: controlled studies find that fluent explanations raise reliance on wrong answers as well as right ones (Kim et al., CHI 2025, N=308), and that deceptive model explanations can be more persuasive than truthful ones (Danry et al., CHI 2025, 1,192 participants). A prose interface does not just fail to show you the state; it can actively degrade your ability to judge it.

References

• Shneiderman, B. Direct Manipulation: A Step Beyond Programming Languages. IEEE Computer, 1983 (term coined 1982).
• Hutchins, E., Hollan, J., & Norman, D. Direct Manipulation Interfaces. Human–Computer Interaction, 1985.
• Subramonyam, H., et al. Bridging the Gulf of Envisioning. CHI 2024.
• Masson, D., Malacria, S., Casiez, G., & Vogel, D. DirectGPT: A Direct Manipulation Interface to Interact with LLMs. CHI 2024.
• Nielsen, J. AI: First New UI Paradigm in 60 Years. Nielsen Norman Group, 2023.
• Maes, P. Agents that Reduce Work and Information Overload. CACM, 1994.
• Shneiderman, B., & Maes, P. Direct Manipulation vs. Interface Agents. interactions, 1997.
• Horvitz, E. Principles of Mixed-Initiative User Interfaces. CHI 1999.
• Victor, B. Inventing on Principle. CUSEC, 2012.
• Haigh, T., & Ceruzzi, P. E. A New History of Modern Computing. MIT Press, 2021.
• Moran, K., & Gibbons, S. Generative UI and Outcome-Oriented Design. Nielsen Norman Group, 2024.
• Morris, M. R. Prompting Considered Harmful. CACM, 2024.
• Zamfirescu-Pereira, J. D., et al. Why Johnny Can’t Prompt. CHI 2023.
• Riche, N., et al. AI-Instruments: Embodying Prompts as Instruments. CHI 2025.
• Zindulka, T., et al. Exploring Mobile Touch Interaction with Large Language Models. CHI 2025.
• Cao, Y., Jiang, P., & Xia, H. Generative and Malleable User Interfaces. CHI 2025.
• Wang, B., et al. LAVE: LLM-Powered Agent Assistance for Video Editing. IUI 2024.
• Dinakar, K., Lieberman, H., & Wu, M.-H. MIND: Blending Agents and Direct Manipulation for Harnessing LLMs. IUI 2025.
• Kim, S. S. Y., et al. Fostering Appropriate Reliance on LLMs. CHI 2025.
• Danry, V., et al. Deceptive Explanations by LLMs. CHI 2025.
• Ink & Switch. Malleable Software. 2025.
• Matuschak, A., & Nielsen, M. How Can We Develop Transformative Tools for Thought?
• Google Research. Generative UI. 2025.