AI Engineer Summit 2023
Shift Left: How to Become an AI Engineer from a Full-Stack Background
Loading slides…
Read the talk
Shift Left: From Full-Stack Development to AI Engineering
A practical learning sequence for full-stack developers: understand language models, learn prompting, compose applications, evaluate changes, and fine-tune when the task warrants it.
From a talk by Reid Mayo
Before you start: You should be comfortable building infrastructure, databases, backend services, and client applications; no prior AI or machine-learning experience is assumed.
What do you need beyond full-stack skills?
If you can build an application from infrastructure through its client interface, what else do you need to build useful AI products? Reid Mayo’s curriculum starts with that question. It assumes you already understand infrastructure, databases and persistence, backend services, and client applications, but have no AI or machine-learning background. The proposed path is a boot camp: a sequence of concepts and practical tutorials aimed at professional AI engineering.
The opportunity comes from foundation models. Shawn “swyx” Wang’s The Rise of the AI Engineer, which inspired the talk’s name, describes how developers can now build useful AI solutions without first acquiring the traditional ML expertise or making the substantial upfront investment in data collection those solutions once required. Full-stack experience is a starting asset: the new work extends an existing ability to turn components into usable products.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a foundation without getting distracted
Before choosing tools, establish a way to learn. Drawing on The Art of Learning, Mayo recommends limiting distractions and avoiding low-information material with diminishing returns. Spend that attention on fundamental building blocks: understanding them thoroughly makes it possible to compose more sophisticated products later.
Use ChatGPT as a private tutor while reading. With Socratic questioning, an unfamiliar term becomes the beginning of a conversation: ask what it means, why it works, and how it relates to concepts you already understand. Mayo invokes a January 2022 knowledge cutoff to emphasize how much foundational material predates the tool; that is his historical reference, not a current cutoff specification. The recording supplies the sequence and the reasons for studying each topic; the assigned materials supply the detailed practice.
Begin with a bounded overview of what large language models are and how they work. Mayo selects Cohere’s educational documentation, noting its connection through a founder to the creation of the transformer architecture. Read module one in its entirety, rather than following every possible branch of the documentation, and continue questioning your tutor as concepts arise. This first pass should give subsequent experiments a conceptual foundation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn prompting before fine-tuning
Prompt engineering can feel strange to someone accustomed to symbolic code and explicit control flow. Why should phrasing a request politely change the quality of a result? Mayo uses that counterintuitive example to introduce the difference between programming conventional logic and interacting with a neural model. The wording of an input can affect output quality, so learning how to construct that input is part of engineering the system.
Skipping directly to fine-tuning misses two benefits. First, the best-performing prompts can become part of the training data used for fine-tuning. Second, hands-on prompting exposes what a model can do and where it fails. Prompting is both an optimization technique and a way to investigate model behavior.
Work through the materials in sequence:
- Watch Elvis Saravia’s overview, then read the Prompt Engineering Guide in full.
- Pay particular attention to the graduate job classification case study. Mayo highlights how successive prompt modifications improve aggregate output quality; the useful lesson is to examine the accumulated effect across examples, rather than one appealing response.
- Read LearnPrompting.org in full. Mayo describes it as favored by OpenAI and recommends the overlap as reinforcement, alongside the additional concepts it introduces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Get to working experiments with OpenAI
OpenAI is the next stop because it combines capable models with accessible interfaces. That combination lets you explore what is possible and begin building quickly. Read the documentation and API reference thoroughly; practical limitations of depending on one provider come later in the curriculum, after you have learned how to use it.
Then briefly survey the hands-on examples in the OpenAI Cookbook. The goal at this point is familiarity with the available patterns, not mastery of every notebook. Return to individual examples when a project needs them. The linked Cookbook and the Evals repository introduced later are evolving collections, so their present contents should not be treated as frozen copies of the materials assigned in the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose an application with LangChain
A model call is only one component of an AI application. LangChain enters the syllabus as a framework for organizing the surrounding pieces into a maintainable, modular system. Those pieces include models, prompts, long- and short-term memory, retrieval-augmented generation, and conversations. Its integration points also allow unsupported or proprietary components to join the application.
That breadth gives the framework a second role: it becomes a map of the AI application ecosystem. Learning how its components fit together exposes more of the engineering practice than studying a model API alone. Start with CommandBar’s nontechnical executive summary, then the plain-English technical guide Mayo recommends for a few basic building blocks. Move from that initial picture of composition into the documentation and implementation.
Read both the Python and JavaScript/TypeScript documentation. At the time of the talk, their coverage differed, so each version offered concepts absent from the other; the repetition also reinforced the shared ideas. As you encounter a documented feature, inspect its implementation on GitHub. This connects the framework’s public abstractions to the code that makes them work.
Finish with Mayo Oshin’s LangChain beginner tutorial to see the fundamentals used in an application. His other walkthroughs then apply those same fundamentals to more complex tasks. The progression is deliberate: understand the pieces, inspect their implementation, and watch them compose into a working system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Establish the test before changing the model
For a full-stack developer, evaluations are the analogue of software tests. Before fine-tuning a black-box model, establish a repeatable way to tell whether a change improves behavior or causes regressions. Without that feedback loop, a different output is easy to mistake for a better one.
A classification task makes the basic structure concrete. The following TypeScript example compares baseline and candidate predictions against the same labeled cases. The records are a teaching example: the candidate fixes one case and breaks another, so an unchanged aggregate score would conceal a meaningful change in behavior.
typescript
type Label = "graduate" | "not-graduate";
type EvalCase = {
id: string;
expected: Label;
baseline: Label;
candidate: Label;
};
const cases: EvalCase[] = [
{
id: "entry-level-developer",
expected: "graduate",
baseline: "not-graduate",
candidate: "graduate",
},
{
id: "senior-engineer",
expected: "not-graduate",
baseline: "not-graduate",
candidate: "graduate",
},
];
const results = cases.map((item) => {
const before = item.baseline === item.expected;
const after = item.candidate === item.expected;
return {
id: item.id,
baselineCorrect: before,
candidateCorrect: after,
change: before === after
? "unchanged"
: after ? "improvement" : "regression",
};
});
console.table(results);
For less constrained outputs, deciding what counts as success requires more care than exact label matching. Mayo explicitly calls out the creativity needed to design effective AI evaluations.
Use the Cookbook’s example evaluations to learn how to write them, then review OpenAI Evals, which provides an evaluation framework, a suite of evaluations, and support for custom ones. This is a focused pass through the materials, but it comes before fine-tuning for a reason: the measurement process needs to exist before the model changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ship, collect data, then test specialization
Now work step by step through OpenAI’s hands-on fine-tuning Cookbook material. Knowing how to adapt its models can take a product a long way, but relying on a hosted provider also introduces production constraints: cost, latency, rate limits, privacy, and control. These constraints supply a practical reason to investigate other models.
Mayo proposes a conditional deployment sequence:
- Prototype and ship with OpenAI’s models to get a useful solution into use quickly.
- Collect usage and training data from that solution.
- When scaling warrants it, evaluate a smaller model fine-tuned for the target use case. Test whether it can match or outperform the original model while costing less.
The smaller model is a candidate to evaluate, not an automatic replacement. The target is performance on the product’s actual task.
After completing the OpenAI material, move to Anyscale’s tutorial on fine-tuning Meta’s Llama 2. Its fine-tuning case study illustrates the task-specific nature of this strategy: fine-tuned models exceeded GPT-4 on some structured tasks, but still trailed it on GSM8k mathematics. Specialization can close a gap without establishing general model parity.
Mayo then cites an OpenPipe case study in which a smaller fine-tuned Llama 2 matched the reported results of OpenAI’s state-of-the-art model for $19 versus approximately $24,000 on the same example task. He describes the example as not cherry-picked, but the talk does not specify the task, quality metric, workload, OpenAI model, or cost accounting. Treat those figures as his reported case-study comparison, not as a cost estimate for a new application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Practice before going deeper
At this point, pause the curriculum and use the skills in the real world. Mayo recommends deploying applications before moving into advanced study. The boot camp has covered enough to build, measure, and adapt an AI system; further reading can follow experience with those responsibilities.
When you are ready for deeper theory, continue with fast.ai’s practical deep learning course and Hugging Face’s NLP course and documentation. These extend the path beyond application composition into a richer understanding of deep learning, more advanced fine-tuning, and eventually training models from scratch. Mayo closes by pointing to the companion syllabus displayed beside him—the path continues, but the next step is to put the fundamentals to work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Swyx's 2023 essay about the engineering discipline emerging around foundation models and AI APIs.
Guides, papers and notebooks for studying prompting techniques and applications.
Practical examples for building with OpenAI models; the collection has evolved since the talk.
An open-source framework and benchmark registry for evaluating language models and building custom evaluations.
A 2023 fine-tuning study comparing Llama 2 on functional representations, SQL generation and grade-school mathematics.
Further reading
A study of prompting language models to identify job postings suitable for graduate or entry-level applicants.
Full transcript
Loading transcript…