AI Engineer World's Fair 2024
AI Engineering Without Borders
Loading slides…
Read the talk
AI Engineering Without Borders
AI engineering needs useful categories, but its durable foundations lie in human constraints, changing model capabilities, and turning those capabilities into tools people can use.
From a talk by swyx
A piece of paper determines whether you can cross
Without the right piece of paper, you cannot cross a line in the sand. The border is a human convention, but its consequences are real: documentation determines where a person can go. That tension opens swyx’s keynote—boundaries can govern our lives without being inherent properties of the world.
AI crosses several boundaries that people treat as separate domains. In swyx’s account, training on predominantly English text can still produce multilingual capabilities; the corpus need not be evenly divided among languages. Modality boundaries are similarly permeable. He cites turning Llama into a vision-language model with roughly $100 of post-training, without specifying the model version, training setup, cost scope, or resulting quality. The point is that a model’s initial modality need not permanently define what it can become.
Other boundary crossings are less welcome. Fluent generation does not reliably distinguish a hallucination from information memorized in a model’s world representation. Copyright is another boundary that model behavior does not automatically respect, although the keynote leaves that topic aside. Crossing boundaries creates both capabilities and engineering obligations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What does an AI engineer do?
For someone identifying as an AI engineer—or trying to hire one—a definition has practical value. One familiar definition draws a line at the model API: it helps separate the responsibilities of people building models from those applying them. That boundary can organize a team even if it does not fully describe the discipline.
There are other useful ways to organize the work. At an earlier AI Engineer Summit, swyx distinguished the AI-enhanced engineer, the AI products engineer, and the non-human agentic AI engineer. A hiring discussion can instead start from job descriptions, as in Elicit’s conversation on How To Hire AI Engineers.
| Definition | What it organizes |
|---|---|
| API boundary | Where team responsibilities meet |
| Engineer subtypes | Different relationships between AI and engineering work |
| Job descriptions | Roles and expectations within a team |
These definitions answer different practical questions; none necessarily exhausts what AI engineering can become.
A workable definition is not yet a shared foundation. Stopping at “you have your perspective, I have mine” leaves no common basis for deciding which claims hold across teams and viewpoints. The next question is what remains true when the organizational boundaries change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
As the field matures, its concerns multiply
Conference programming makes these definitions concrete. In a conversation with Raza Habib, swyx described AI Engineer conferences as his highest-stakes expression of what the field currently is. Earlier tracks covered retrieval-augmented generation (RAG), code generation, agents, and multimodality. Those topics remain, but they no longer encompass all the concerns a competent practitioner must hold together.
The analogy is Inside Out: maturity adds more concerns to juggle. Once an application works, the engineer may need to migrate to open models, build evaluations—and reconsider whether those should have come first—scale inference, or deploy into a Fortune 500 organization. These are overlapping responsibilities, not successive replacements for the original application techniques.
Management adds hiring AI engineering teams and setting company AI strategy. Alongside the technical and organizational tracks sits the community itself. Its value cannot be established just by advertising that a conference has a good network; people have to experience it. Shared photos and posts also let those outside the venue participate, including swyx, who is addressing the gathering remotely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tracks are invented; constraints are discovered
The difficulty with treating those tracks as fundamental divisions is knowing how they were made. Looking back at The Rise of the AI Engineer on its first anniversary, swyx finds a list of disciplines that largely became the topics of his conferences and meetups. The categories began as an organizing choice.
Why separate agents from code generation, when an agent can generate code? Why separate RAG from open models, when a system can use both? A competent AI engineer needs familiarity across these areas. Conference organizers draw one set of boundaries; employers draw another. Neither set obliges reality to behave accordingly.
Physical regularities offer a different standard. An alien civilization arriving on Earth would discover the same gravity, solar energy, and magnetic field, regardless of its institutions or preferences. That thought experiment supplies the test: which foundations of AI engineering would independent practitioners discover again, even if they started with different categories?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Engineering connects constraints to human benefit
To look for those foundations, swyx compares definitions of software engineering with broader definitions of engineering. The software definitions he attributes to IEEE and Google Developers emphasize design, implementation, testing, documentation, and lifecycle management. The broader engineering definitions he cites emphasize applying natural sciences for humanity’s benefit. His criticism concerns the emphasis of those particular software definitions, rather than establishing that software engineering as a profession lacks obligations to people.
AI engineering sits somewhere between these descriptions: software practice encountering more of the real world’s constraints than a typical software engineering career might demand. The resulting question is not merely how to implement a system, but what constraints govern it and what human purpose it serves. The proposed “laws” that follow are starting points for debate, not a complete theory with the status of physics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Human rates and temporary technology baselines
Start with the person using the system. Swyx uses 80 spoken words per minute versus 200 read words per minute as a design heuristic, not a universal human limit. The disparity matters: speech and text interfaces have different throughput needs because people consume them differently. A generation rate that suits one interaction mode need not suit another.
Then separate relatively stable human constraints from contingent facts: things that are true for now. Apple Intelligence supplies the example. Swyx cites Apple’s advertised 30 output tokens per second for local inference. Apple’s foundation-model report places that historical measurement on an iPhone 15 Pro, before speculative decoding; it is generation throughput, not prompt-processing speed. His broader argument is conditional: if local models become widely available on phones, their performance establishes a practical baseline for intelligence cheap enough to treat as an everyday utility.
These baselines also change. Swyx adds trends as a third category, invoking Moore’s law as an analogy for planning around sustained technical progress. Unlike gravity, these are not physical laws. Their usefulness lies in helping an engineer avoid bets that depend on an obvious, improving limitation remaining fixed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Plan for better context use and cheaper intelligence
Context is the first trend. Swyx recalls discussing MPT-7B with Mosaic a year earlier: roughly 60,000–70,000 tokens of context, with what he describes as substantial loss. The long-context example corresponds to the specialized MPT-7B-StoryWriter-65k+ fine-tune, rather than the base model generally. He contrasts that earlier capability with people in the audience who have trained million-token context windows.
Context length and context utilization are different capabilities. A larger window tells you how much material can be supplied; it does not establish how reliably the model uses that material. Swyx points to the recently released Claude 3.5 Sonnet as evidence of improving utilization. His description of complete utilization is an assessment of that progress, not a guarantee that every token will be used correctly across tasks. The engineering trend to watch is therefore both capacity and effective use.
Cost is another changing constraint. Swyx reports a 99.55% decline in the cost of GPT-3-level intelligence over the preceding two years. He tentatively estimates an 80–90% decline for GPT-4-level intelligence, referencing GPT-4, Llama 3, and newer models. These are historical comparisons without a specified capability benchmark, workload, pricing basis, or serving conditions. They support his commodification argument, but do not supply enough detail to calculate a particular application’s savings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Utility needs its own dimension
Where does this leave AI engineering among competing AI philosophies? A line between effective altruism (EA) and effective accelerationism (e/acc) might place engineers somewhere in the middle: concerned about safety, while also wanting progress. But that single axis does not express the purpose of the work.
Swyx proposes a second dimension: utility. His phrase is “utility maxis”—people who look at available capabilities and want to use them for humanity’s benefit. Safety and acceleration remain concerns, but neither alone answers what useful thing an engineer will build.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From an untamable monster to mass transit
Working across boundaries begins with questioning the ones already in your head. Avoid personal dogma, easy consensus, and passive reactions. Disagree with your own conclusions, with peers productively, and with the status quo. That makes a conference more than a social occasion: it can become a point at which an engineer’s understanding and career change direction.
The closing image is a Shoggoth, the monster used as a metaphor for an unfamiliar model beneath its friendly interface. An AI engineer looks at it and sees the possibility of mass rapid transit. The ambition is to turn a difficult force into a tool that people can use, rather than stopping at the judgment that it cannot be tamed.
That ambition requires movement between the categories people have created. Jump between conference tracks, friend groups, disciplines, and modalities. A gathering like this offers room to make connections outside the boundaries of a job—and to discover what becomes possible when those boundaries no longer determine whom you learn from or what you build.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original essay proposing AI engineering as a discipline and outlining its emerging tools and responsibilities.
Elicit's discussion of AI engineering skills, team roles and hiring, with a companion podcast and hiring materials.
Apple's technical overview of its initial on-device and server models, including iPhone inference optimizations and measured generation speed.
MosaicML's original model-family announcement explains the long-context StoryWriter variant and its training setup.
The historical announcement for the first Claude 3.5 Sonnet release.
Further reading
A meta-analysis of silent and oral reading rates, including variation across readers and tasks.
- Software Engineering Code of EthicsDocumentation
The IEEE-CS/ACM professional code places public interest, safety and welfare among software engineers' obligations.
Full transcript
Loading transcript…