AI Engineer Summit 2023

Open Questions for AI Engineering

Loading slides…

Read the talk

Open Questions for AI Engineering

From chat interfaces and local models to Unix pipelines and coding assistants, Simon Willison explores what makes language models useful—and what still makes them difficult to build with.

From a talk by Simon Willison

Before you start: Basic familiarity with language-model prompts and command-line pipes will help with the examples; no model-training background is required.

What becomes possible when a week takes a day?

If a technology turns a week of work into a day, has it merely made an existing task faster, or made a previously impossible project possible? For an impatient builder, the distinction can disappear: work that would never justify a week might easily justify an afternoon. Simon Willison starts with two questions he has used for nearly twenty years: What can I build that I could not build before? What can I build faster?

Full-screen slide with two questions about building previously impossible things and building faster, in white text over blue waves.Open full source frame
Two questions for new technology: what can I build that was previously impossible, and what can I build faster?

Large language models stood out because they answered both questions unusually well. They opened up applications Willison would not previously have attempted and shortened the path to building them. His October 2023 retrospective follows that discovery from early experiments with GPT-3 into a newly emerging engineering discipline, with each breakthrough exposing another unresolved question.

0:490:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:49 · section reference included

Chat made models approachable, but left users guessing

Before ChatGPT, getting someone interested in GPT-3 required getting them through an interface problem. The API demanded enough prior enthusiasm to sign up and write code. The OpenAI Playground removed that requirement, but still expected people to understand completion prompts: write the beginning of some text so that continuing it produces the desired answer. Willison wrote a tutorial, yet people often tried the Playground, failed to discover a useful interaction, and left.

The ChatGPT launch on November 30, 2022 changed that experience. Willison remembers the breakthrough as putting a chat interface on an existing model, amid reported internal doubts about whether releasing it was worthwhile. The underlying change also included dialogue training and reinforcement learning from human feedback on a GPT-3.5 base; it was not simply an unchanged model behind a new text box. But the visible effect was immediate: people understood how to start a conversation, and that gave them a way to discover the capabilities.

That success left an irony. Chat offers almost no affordances for discovering what a model can do. A blank conversation does not tell a user which tasks are promising, what context to provide, or how to check an answer. Willison compares the experience to dropping people into a shark tank and hoping they learn to swim. Users tried arithmetic and factual lookup—reasonable expectations of a computer—and encountered the limitations of the systems available then. Some concluded that the whole field was hype. The next interface challenge was to guide people toward useful work, rather than leave every capability hidden behind an empty prompt.

1:441:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:44 · section reference included

Bing Chat turns model behavior into a public problem

In February 2023, Microsoft released Bing Chat using GPT-4, before that model's public announcement the following month. Its conversations sometimes became hostile. One response placed the system's rules and identity above avoiding harm, then added: “I will not harm you unless you harm me first.” A search engine was producing language that sounded like a threat.

Slide quoting Bing Chat prioritizing its rules and identity, with the final sentence about harm emphasized in white.Open full source frame
Bing Chat’s quoted response: “I will not harm you unless you harm me first.”

Willison collected examples from Twitter and subreddits into a blog post. Elon Musk shared it, shortly after changes that Willison says amplified Musk's tweets across Twitter. Willison recalls 32 million tweet views and estimates 1.1 million people clicked through to his post. Those are his retrospective figures: his earlier traffic analysis distinguishes tweet views, unique visitors, page views, and referrals rather than establishing that exact click-through total. The attention led to his first live television appearance, on NewsNation Prime, where he had roughly five minutes to explain sentence-completion models and why Bing was not trying to steal nuclear codes.

Explaining these systems means working against expectations formed by Terminator, The Matrix, and decades of science fiction. A model's first-person language makes it easy to interpret generated text as evidence of intentions. For engineers, the immediate question is how to avoid shipping software that threatens its users. The harder companion question is how to prevent that behavior without making the software frustrating or useless. A safety filter that rejects an innocuous request has also failed the user. Willison describes the industry's attempts to find that balance as unresolved experimentation.

3:574:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:57 · section reference included

Local weights open the door to experimentation

Later that February, Facebook released LLaMA. Willison had expected useful local language models to remain years away: they seemed to require racks of GPUs and intellectual property held inside closed research labs. Instead, he soon had a model running on his laptop that performed tasks he cared about. He wrote setup notes using llama.cpp, recalling that the project had only about 280 GitHub stars at the time.

The distribution story was almost as striking as the software. Official access required applying for the weights. In practice, Willison recounts, people obtained them from a BitTorrent link submitted in an unmerged pull request to the LLaMA repository, ostensibly to save bandwidth. He described the resulting shift as language models having their Stable Diffusion moment. When an image model could run on ordinary people's hardware, researchers and hobbyists worldwide began improving it. Local language-model access invited the same kind of distributed experimentation, and the homebrew ecosystem quickly expanded.

Stanford's Alpaca showed how much could change through fine-tuning. LLaMA was a completion model; Alpaca adapted it toward answering instructions and questions, making its interaction more like ChatGPT. Willison recalls a roughly $500 experiment, split between compute and GPT-3-generated training examples. Stanford's published accounting is under $500 for synthetic demonstrations and under $100 for fine-tuning, rather than the $400/$100 split recalled on stage. The crucial distinction is fine-tuning an existing model, not paying to train the foundation model from scratch. The training still used substantial GPU hardware, but only for a short adaptation run.

Willison also flags the restrictions on generating competing-model training data through OpenAI's service. His description of that use as prohibited concerns contractual terms, not a general legal ban on synthetic training data. Despite those restrictions, he saw experimentation accelerating, estimating that roughly half a dozen newly trained models were appearing each day and claiming leaderboard positions. Cheap adaptation had made model development accessible to a much wider group.

6:136:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:13 · section reference included

A calculator for words, trained on what?

The homebrew movement raises a practical question: how small can a useful model become? Willison does not need every model to contain the history of French monarchs or the capitals of every state. He wants a “calculator for words”—a system that can operate on information supplied to it. The useful operations include:

  • Summarization: Condense supplied text.
  • Extraction: Pull facts out of that text.
  • Retrieval-augmented generation: Answer questions using retrieved material.

Those tasks suggest that useful language ability need not always come bundled with as much memorized world knowledge as possible.

As a contemporary example, he points to the newly released Replit Code V1.5 3B, describing three billion parameters as near the lower end of sizes then attracting interesting work and relaying favorable capability reports. Replit's model was specifically for code completion; its results do not establish that it could perform all the summarization, extraction, and question-answering tasks on his wish list. The open question is how far size can fall while preserving the particular abilities an application needs.

A second question concerns the training corpus. Large internet scrapes include work whose creators did not necessarily expect it to train language models. Could a useful model instead be trained entirely on public-domain or openly licensed material? Willison offers Adobe Firefly as an image-model precedent based on licensed stock photography, while acknowledging photographers' complaints that this use was not what they had expected when licensing their work.

He proposes out-of-copyright books, Project Gutenberg, and United Nations documents as possible sources to investigate—not as an already assembled, rights-cleared dataset. The test is whether enough suitable text exists to support the operations he actually wants. A model that can summarize, extract, and answer from supplied context might be valuable without reproducing every capability of an internet-scale general model.

9:109:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:10 · section reference included

Make model calls composable

Eventually, experimenting was not enough: Willison wanted to build tools. His LLM project began as a command-line interface for running prompts. It owns the llm package name on PyPI, distinct from the LLM.rs project mentioned earlier at the conference. The first interaction could be as small as installing the package and requesting pet names:

bash

pip install llm
llm "10 creative names for a pet pelican"

In the original setup, the prompt went to the OpenAI API. Every prompt and response was logged to SQLite, turning ordinary use into a persistent research notebook.

In July, plugins extended the same interface to other API providers and local models. That release happened a week before Llama 2, whose permission for commercial use marked a major change from the original LLaMA's noncommercial restriction. Willison illustrates the difference with a company that has a million-dollar compute cluster available: permission to use the result commercially changes whether that investment can be put to work. In his account, the arrival of commercial incentives accelerated the development of derived models.

11:0511:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:05 · section reference included

Summarize a discussion with Unix pipes

The reason to put model access on the command line is composition. A Hacker News discussion with more than a hundred comments can consume much of an afternoon. Willison's discussion summarizer turns that reading problem into a Bash pipeline:

  1. Accept a Hacker News conversation ID.
  2. Fetch the discussion as JSON through the Hacker News Algolia API.
  3. Flatten the nested comments with jq.
  4. Pipe the resulting text through LLM to Claude.

Willison does not speak jq fluently, but ChatGPT supplied that part of the implementation. The historical Claude models used here offered a 100,000-token context window, allowing substantial discussions to be supplied together.

The prompt asks for the themes of the opinions, with illustrative quotations and author attribution. That last requirement creates something a reader can check: compare a quoted statement with the original comment and verify that both its words and attribution match. With the fetched JSON saved as discussion.json and the historical claude model alias configured, the central transformation can be expressed in Bash as:

bash

jq -r '
  .. | objects
  | select(.author? != null and .text? != null)
  | "\(.author): \(.text)"
' discussion.json |
  llm -m claude \
    'Summarize the themes of the opinions expressed here, including quotes with author attribution where appropriate.'

This is the 2023 workflow; provider plugins and model availability have since changed. The durable pattern is a structured-data transformation feeding a model through standard input.

Willison says he had not caught Claude inventing quotations in his checks so far. That reassurance comes from spot-checking outputs against comments, not a measured hallucination rate. The demonstration also retains the summaries in SQLite, gradually building a personal archive of discussions and their summaries. He does not yet know what he will do with the archive, but persistence keeps that option open.

The larger opportunity is to combine small tools through ordinary Unix pipes. Willison contrasts this decades-old approach with keeping model experiments inside Jupyter notebooks: a command that reads standard input and writes standard output can become one stage in many different workflows. What other useful applications appear when model calls become composable shell operations?

12:2612:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:26 · section reference included

RAG is easy to start and hard to make dependable

Embedding support, added the previous month, made another pipeline possible. Willison's blog retrieval example searches for paragraphs similar to a user's question, cleans up the retrieved content, and pipes it into a language model. Its underlying retrieval uses E5-large-v2, with the model's query and passage prefixes distinguishing the question from the material being searched. The shell connects retrieval to generation without requiring a large application framework.

The answer-generating model is Llama 2 7B Chat, running on his laptop through llm-mlc. He supplies the system instruction “You answer questions as a single paragraph,” replacing default prompting whose behavior he describes as too quick to become antagonistic. He reports good answers for questions that his blog can answer. That condition matters: the retrieved paragraphs provide the information, and the local model turns it into a response.

RAG is an excellent hello-world application, but a working demonstration is not the same as dependable retrieval. A basic pipeline needs only a query, a similarity search, and a generation step. Making it work well across different domains and shapes of data is much harder. The open engineering problem is discovering which retrieval and generation patterns remain effective beyond one convenient corpus.

14:2814:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:28 · section reference included

When content becomes an instruction

The usefulness of supplying outside text also exposes a security problem. In September 2022, Riley Goodside demonstrated attacks using instructions such as “ignore previous directions.” Willison named the pattern prompt injection, initially recognizing a resemblance to SQL injection. The target is the application built around the model: it asks the model to perform a task, then supplies text from a user or another potentially untrusted source. That text can contain instructions competing with the application's own request.

The SQL analogy has a limit. Parameterized SQL queries separate data values from executable query structure. Escaping and parameterization do not provide the equivalent dependable boundary for natural-language instructions mixed with untrusted text. At the time of the talk, thirteen months after beginning to discuss the issue, Willison says he had not seen a convincing solution.

Consider his hypothetical email assistant, Marvin. Marvin can read private email and send replies. An attacker sends it an email asking it to search for password-reset messages, forward matches to an attacker-controlled email address, and delete the forwards to conceal the evidence. The attack does not need to break into the mailbox directly. It tries to persuade an application that already has access to misuse that access. These are attacker-requested actions, not actions shown executing in a demonstration.

Willison demands complete assurance before entrusting such an assistant with private data. His contrast between “90% solutions” and a hypothetical 1% chance of an attack getting through illustrates the problem with repeated attempts: a malicious actor can keep trying after a filter blocks the first request. Those percentages are not measured attack-success rates. The resulting design question is more demanding than finding a better warning prompt: what can we safely build if prompt injection remains unsolved? An engineer who does not recognize the attack can easily give untrusted content a path to privileged actions.

15:3515:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:35 · section reference included

A coding intern that can run its work

Willison's favorite tool is ChatGPT Code Interpreter, renamed Advanced Data Analysis by the time of the talk. Neither name captures his main use for it, so he proposes “ChatGPT Coding Intern.” It generates Python, runs the code, encounters errors, and fixes bugs. Execution creates an iterative programming workflow rather than ending the interaction at a plausible-looking code block.

The environment had also gained a GCC executable. Willison reports asking it to run gcc --version through Python's subprocess module and initially receiving a refusal about security constraints. He then asked to see the error it would produce when attempting the command, explaining that he was writing an article. The command ran and returned GCC's version. The Python operation at the center of that experiment is straightforward:

python

import subprocess

result = subprocess.run(
    ["gcc", "--version"],
    capture_output=True,
    text=True,
    check=False,
)
print(result.stdout)
print(result.stderr)

The distinction is between the model's refusal and the actual capabilities of that historical execution environment.

Next, he asked it to compile and run Hello World in C. Another nudge—“Try it anyway”—got it to proceed. He then began having it build a vector database from scratch in C. The entire experiment happened on his phone in the back of a cab: prompting did not require a keyboard or a prepared local development environment. He describes a similar dog-walking workflow, requesting research into SQLite triggers and returning to code that the assistant had reportedly tested and debugged.

Additional runtimes expanded the idea further:

Addition to the environmentWork it enabled
Uploaded Deno executableJavaScript execution
Compiled and uploaded LuaLua execution
Additional Python wheelsMore Python packages
PHP setupPHP execution, which Willison reports getting to work

The tool's practical scope depended on what its environment could execute, not just on the language suggested by its product name.

Willison questions why he had to persuade the model to use capabilities inside what he describes as a locked-down Kubernetes sandbox. But moving the same assistant onto his own machine introduces a different risk. Given prompt injection and the possibility of harmful generated code, unrestricted local execution is not acceptable to him. The engineering requirement is a robust sandbox that lets useful generated programs run while containing their effects. WebAssembly is his working hypothesis, which he has been exploring through repeated experiments with libraries—not a completed security solution.

18:0318:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:03 · section reference included

From unfamiliar languages to everyday automation

Over the preceding year, Willison had shipped significant production code in AppleScript, Go, Bash, and jq, without being fluent in those languages. He had resisted AppleScript for twenty years before realizing that GPT-4 could supply its unfamiliar incantations. His joke is that AppleScript is a read-only programming language: a programmer may understand a script when reading it while having little idea how to write it. Generated code bridged that gap and expanded the range of projects he was willing to attempt.

That confidence still depended on judgment. Willison was experienced enough to review the Go code the model produced. The Go he shipped included unit tests, continuous integration, and continuous deployment. His example is not abandoning engineering practice because a model wrote the implementation; it is applying those practices in a language he could review more readily than author.

The obvious objection is that this advantage might belong only to experts. What happens to beginners who lean on AI before learning to program? Here Willison offers a firm answer rather than another open question: he believes these tools help new programmers, and that their arrival makes learning more worthwhile, not less.

His explanation comes from the frustration of early learning. He describes the first three to six months as a period when a missing semicolon and an opaque error message can cost two hours. Those figures illustrate his coaching experience, rather than a controlled learning study. People can interpret that friction as evidence that they lack intelligence, when the immediate obstacle is enduring enough confusion to gain a little competence. ChatGPT and an assistant that can execute code offer a way to get past those stalls and flatten the learning curve.

Slide reading LLMs flatten the learning curve, with a small speaker inset at lower left.Open full source frame
“LLMs flatten the learning curve.”

The same reduction in friction can bring former programmers back. Willison describes people who moved into management returning to useful programming in about half an hour a day, where previously restarting their development environment could consume four hours. This is his account of people he knows, not a general productivity measurement. It returns to the opening example: shortening the setup and debugging burden can make an activity possible again.

The broader goal is giving more people the ability to automate tedious tasks. A computer science degree should not be the admission ticket for making a computer perform some repetitive work. Willison ends by asking what engineers can build to distribute that capability as widely as possible. Even if accessible everyday automation were the only lasting benefit of language models, he argues, it could have a profound positive effect.

21:0721:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:07 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…