AI Engineer World's Fair 2024

AI Engineering Without Borders

swyx· Editor, Latent.Space10:32

Loading slides…

Read the talk

AI Engineering Without Borders

AI engineering needs useful categories, but its durable foundations lie in human constraints, changing model capabilities, and turning those capabilities into tools people can use.

From a talk by swyx

A piece of paper determines whether you can cross

Without the right piece of paper, you cannot cross a line in the sand. The border is a human convention, but its consequences are real: documentation determines where a person can go. That tension opens swyx’s keynote—boundaries can govern our lives without being inherent properties of the world.

AI crosses several boundaries that people treat as separate domains. In swyx’s account, training on predominantly English text can still produce multilingual capabilities; the corpus need not be evenly divided among languages. Modality boundaries are similarly permeable. He cites turning Llama into a vision-language model with roughly $100 of post-training, without specifying the model version, training setup, cost scope, or resulting quality. The point is that a model’s initial modality need not permanently define what it can become.

Other boundary crossings are less welcome. Fluent generation does not reliably distinguish a hallucination from information memorized in a model’s world representation. Copyright is another boundary that model behavior does not automatically respect, although the keynote leaves that topic aside. Crossing boundaries creates both capabilities and engineering obligations.

Dark blue slide titled “AI = Border Disrespecter” lists language, modality, ground truth and copyright beside a small speaker inset.Open full source frame
AI as a “Border Disrespecter”: language, modality, ground truth and copyright.
0:000:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

What does an AI engineer do?

For someone identifying as an AI engineer—or trying to hire one—a definition has practical value. One familiar definition draws a line at the model API: it helps separate the responsibilities of people building models from those applying them. That boundary can organize a team even if it does not fully describe the discipline.

There are other useful ways to organize the work. At an earlier AI Engineer Summit, swyx distinguished the AI-enhanced engineer, the AI products engineer, and the non-human agentic AI engineer. A hiring discussion can instead start from job descriptions, as in Elicit’s conversation on How To Hire AI Engineers.

DefinitionWhat it organizes
API boundaryWhere team responsibilities meet
Engineer subtypesDifferent relationships between AI and engineering work
Job descriptionsRoles and expectations within a team

These definitions answer different practical questions; none necessarily exhausts what AI engineering can become.

A workable definition is not yet a shared foundation. Stopping at “you have your perspective, I have mine” leaves no common basis for deciding which claims hold across teams and viewpoints. The next question is what remains true when the organizational boundaries change.

1:111:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:11 · section reference included

As the field matures, its concerns multiply

Conference programming makes these definitions concrete. In a conversation with Raza Habib, swyx described AI Engineer conferences as his highest-stakes expression of what the field currently is. Earlier tracks covered retrieval-augmented generation (RAG), code generation, agents, and multimodality. Those topics remain, but they no longer encompass all the concerns a competent practitioner must hold together.

The analogy is Inside Out: maturity adds more concerns to juggle. Once an application works, the engineer may need to migrate to open models, build evaluations—and reconsider whether those should have come first—scale inference, or deploy into a Fortune 500 organization. These are overlapping responsibilities, not successive replacements for the original application techniques.

Four topic groups labeled Open Models, Evals, GPUs & Inference, and AI in F500 pair illustrated characters and portraits with company logos.Open full source frame
Conference topics include open models, evals, GPUs and inference, and AI in F500.

Management adds hiring AI engineering teams and setting company AI strategy. Alongside the technical and organizational tracks sits the community itself. Its value cannot be established just by advertising that a conference has a good network; people have to experience it. Shared photos and posts also let those outside the venue participate, including swyx, who is addressing the gathering remotely.

2:472:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:47 · section reference included

Tracks are invented; constraints are discovered

The difficulty with treating those tracks as fundamental divisions is knowing how they were made. Looking back at The Rise of the AI Engineer on its first anniversary, swyx finds a list of disciplines that largely became the topics of his conferences and meetups. The categories began as an organizing choice.

Why separate agents from code generation, when an agent can generate code? Why separate RAG from open models, when a system can use both? A competent AI engineer needs familiarity across these areas. Conference organizers draw one set of boundaries; employers draw another. Neither set obliges reality to behave accordingly.

Physical regularities offer a different standard. An alien civilization arriving on Earth would discover the same gravity, solar energy, and magnetic field, regardless of its institutions or preferences. That thought experiment supplies the test: which foundations of AI engineering would independent practitioners discover again, even if they started with different categories?

A globe above Made up Rules lists borders, rights and money; Earth above Laws of Nature lists gravity, sun and magnetic field.Open full source frame
Made-up rules versus laws of nature.
4:294:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:29 · section reference included

Engineering connects constraints to human benefit

To look for those foundations, swyx compares definitions of software engineering with broader definitions of engineering. The software definitions he attributes to IEEE and Google Developers emphasize design, implementation, testing, documentation, and lifecycle management. The broader engineering definitions he cites emphasize applying natural sciences for humanity’s benefit. His criticism concerns the emphasis of those particular software definitions, rather than establishing that software engineering as a profession lacks obligations to people.

AI engineering sits somewhere between these descriptions: software practice encountering more of the real world’s constraints than a typical software engineering career might demand. The resulting question is not merely how to implement a system, but what constraints govern it and what human purpose it serves. The proposed “laws” that follow are starting points for debate, not a complete theory with the status of physics.

5:496:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:49 · section reference included

Human rates and temporary technology baselines

Start with the person using the system. Swyx uses 80 spoken words per minute versus 200 read words per minute as a design heuristic, not a universal human limit. The disparity matters: speech and text interfaces have different throughput needs because people consume them differently. A generation rate that suits one interaction mode need not suit another.

Then separate relatively stable human constraints from contingent facts: things that are true for now. Apple Intelligence supplies the example. Swyx cites Apple’s advertised 30 output tokens per second for local inference. Apple’s foundation-model report places that historical measurement on an iPhone 15 Pro, before speculative decoding; it is generation throughput, not prompt-processing speed. His broader argument is conditional: if local models become widely available on phones, their performance establishes a practical baseline for intelligence cheap enough to treat as an everyday utility.

These baselines also change. Swyx adds trends as a third category, invoking Moore’s law as an analogy for planning around sustained technical progress. Unlike gravity, these are not physical laws. Their usefulness lies in helping an engineer avoid bets that depend on an obvious, improving limitation remaining fixed.

Two columns compare Laws of Earth—gravity, sun and magnetic field—with Laws of AI Engineering—constants, contingent facts and trends (“Moore’s Laws”).Open full source frame
Laws of AI engineering: constants, contingent facts and trends.
6:587:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:58 · section reference included

Plan for better context use and cheaper intelligence

Context is the first trend. Swyx recalls discussing MPT-7B with Mosaic a year earlier: roughly 60,000–70,000 tokens of context, with what he describes as substantial loss. The long-context example corresponds to the specialized MPT-7B-StoryWriter-65k+ fine-tune, rather than the base model generally. He contrasts that earlier capability with people in the audience who have trained million-token context windows.

Context length and context utilization are different capabilities. A larger window tells you how much material can be supplied; it does not establish how reliably the model uses that material. Swyx points to the recently released Claude 3.5 Sonnet as evidence of improving utilization. His description of complete utilization is an assessment of that progress, not a guarantee that every token will be used correctly across tasks. The engineering trend to watch is therefore both capacity and effective use.

Cost is another changing constraint. Swyx reports a 99.55% decline in the cost of GPT-3-level intelligence over the preceding two years. He tentatively estimates an 80–90% decline for GPT-4-level intelligence, referencing GPT-4, Llama 3, and newer models. These are historical comparisons without a specified capability benchmark, workload, pricing basis, or serving conditions. They support his commodification argument, but do not supply enough detail to calculate a particular application’s savings.

7:568:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:56 · section reference included

Utility needs its own dimension

Where does this leave AI engineering among competing AI philosophies? A line between effective altruism (EA) and effective accelerationism (e/acc) might place engineers somewhere in the middle: concerned about safety, while also wanting progress. But that single axis does not express the purpose of the work.

Swyx proposes a second dimension: utility. His phrase is “utility maxis”—people who look at available capabilities and want to use them for humanity’s benefit. Safety and acceleration remain concerns, but neither alone answers what useful thing an engineer will build.

8:488:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:48 · section reference included

From an untamable monster to mass transit

Working across boundaries begins with questioning the ones already in your head. Avoid personal dogma, easy consensus, and passive reactions. Disagree with your own conclusions, with peers productively, and with the status quo. That makes a conference more than a social occasion: it can become a point at which an engineer’s understanding and career change direction.

The closing image is a Shoggoth, the monster used as a metaphor for an unfamiliar model beneath its friendly interface. An AI engineer looks at it and sees the possibility of mass rapid transit. The ambition is to turn a difficult force into a tool that people can use, rather than stopping at the judgment that it cannot be tamed.

That ambition requires movement between the categories people have created. Jump between conference tracks, friend groups, disciplines, and modalities. A gathering like this offers room to make connections outside the boundaries of a job—and to discover what becomes possible when those boundaries no longer determine whom you learn from or what you build.

9:169:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:16 · section reference included

Resources

From the talk

  • The original essay proposing AI engineering as a discipline and outlining its emerging tools and responsibilities.

  • Elicit's discussion of AI engineering skills, team roles and hiring, with a companion podcast and hiring materials.

  • Apple's technical overview of its initial on-device and server models, including iPhone inference optimizations and measured generation speed.

  • MosaicML's original model-family announcement explains the long-context StoryWriter variant and its training setup.

  • The historical announcement for the first Claude 3.5 Sonnet release.

Transcript download
Full transcript

Loading transcript…