AI Engineer World's Fair 2024
Hiring & Building an AI Engineering Team
Loading slides…
Read the talk
Hiring and Building an AI Engineering Team
Build an AI team around the work the product needs next: production engineering, data judgment, product discovery, and sustained access to domain experts.
From a talk by Bryan Bischof
What does an AI engineer actually do?
What should you hire someone to do when the job title itself is still being defined? At Hex, a data science platform that Bryan Bischof describes as a powerful Jupyter Notebook, that question was already practical. He had spent about a year and a half building AI capabilities, following roughly twelve years in data science and machine learning. His first AI-engineer hiring tweet, he says, helped inspire The Rise of the AI Engineer, though he does not agree with everything in the essay.
To settle the definition, he introduces AI Leader GPT, a small chatbot, and asks what an AI engineer means. It has no useful answer. Bischof turns the failure into a joke about progress on MMLU: apparently, improving benchmark performance has not resolved this particular ambiguity. The practical starting point is the work itself. Building an AI product requires a team; his first job posting explains the contribution he needed from its first AI engineer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The first hire must get capabilities into production
The opening was for a senior engineer, coming from either software engineering or machine learning engineering, who could rapidly expand Hex’s capabilities in greenfield applications. The posting explicitly excluded a research role. That distinction saves both sides time: a candidate hoping to produce work for ICLR should know whether the company can support that ambition. For Hex, this was a constraint of the company’s stage, not a judgment about the value of research.
The positive requirements were experience putting ML or AI capabilities into production for real users, enthusiasm for applying AI to business problems, and an understanding of system architecture. MLOps knowledge or infrastructure-adjacent backend experience could provide that foundation.
The language requirement was deliberately broader than Python: candidates needed to be comfortable working in Python and TypeScript, or strong in one and willing to learn the other. Hex’s application was built in TypeScript, so the AI engineer needed to enter that codebase and work through integration details. React expertise was optional; productive collaboration with the people building the application was not.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match the team to the product’s stage
The next question is when to add each capability. Bischof’s progression starts with a small combination of engineering, data, and product skills, then adds the specialties that a developing product makes necessary.
| Product stage | Team capabilities |
|---|---|
| Early | Software engineering, data expertise, product competency |
| Middle | More software engineering and infrastructure, continued data expertise, design |
| Later | Scaled infrastructure and existing functions, plus specialized ML engineering |
Design is the middle-stage capability he most often sees teams underinvest in.
An audience question makes “data expertise” concrete. Data scientists and analysts notice when a distribution looks strange, when user outputs differ from expectations, or when retention looks weak. The question is not simply whether retention is going up. It is what retention should look like for this product and how comparable products perform. Bischof supplies no target percentage; he recommends hiring people with the experience to investigate those questions.
Later, specialized MLE work becomes easier to justify through a specific need: fine-tuning a model, fine-tuning an embedding model, or improving search and retrieval. These are reasons to add expertise once the product exposes the need for it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hiring must follow the learning sequence
Starting early does not mean staffing heavily. Bischof invokes the Mythical Man-Month to reject a tempting calculation: if an AI demo takes one week, surely twenty engineers can turn it into a product in four weeks. Those numbers describe the assumption he is criticizing, not a delivery forecast. His pointed analogy is that nine women cannot give birth in one month. Some work cannot be compressed by dividing it among more people.
In the early AI-product landscape he describes, this constraint is especially consequential: teams are still discovering what to build. Overstaffing is the leading mistake he reports hearing about from peers. Adding engineers to an uncertain product does not remove the uncertainty that determines their next useful task.
The hiring schedule should reflect the development schedule. His sequence is:
- Put an early product in front of users.
- Build evaluations.
- Gather user feedback.
- Iterate.
He points to the coauthored practitioner work What We’ve Learned From A Year of Building with LLMs and conversations with other builders as support for this order. Hiring should supply the capabilities needed to carry out that sequence.
This also moves data expertise earlier than in a conventional product-engineering effort. Everyone can inspect outputs, but professional analysts have accumulated intuition about what deserves a second look. That experience helps a team extract useful information from its first users and evaluations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give every role a hiring thesis
Why hire instead of using people already at the company? The answer needs to be a capability that the team lacks. If an internal transfer can provide it, Bischof sees no reason to insist on an external hire. For the first full-stack engineer, that thesis is straightforward: integrate the application with an LLM provider and build the minimum infrastructure.
The next roles address different gaps:
- Data scientist: Own evaluation, quality, and user-data analysis so the product can improve continuously.
- Product specialist: Talk to users and understand their jobs to be done. The person needs a strong product capability, but not necessarily a product-manager title.
- Designer: Help discover the shape the experience should take while AI interface conventions are still unsettled.
Not knowing the application’s jobs to be done is already a hole in the team. Treating design as a finishing step leaves another one.
Bischof illustrates the design gap with his own chatbot. Its React interface was written by Claude, and he offers it as an example of how immature AI applications can look. He extends the criticism to ChatGPT: a familiar chat interface is not necessarily the endpoint of good AI-product design. His recommendation is to bring in a professional who can help discover a better experience.
The MLE’s hiring thesis comes last: push the product beyond commodity intelligence. Provider integration gets a team access to broadly available model capabilities. Specialized ML work should create the additional capability the product needs beyond that baseline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look for judgment beyond completing the implementation
Data intuition shows up in what an engineer notices after an operation succeeds. Creating semantic embeddings for every document is one accomplishment. Inspecting their mutual distances and noticing a ridge-like, jagged structure is another. Bischof treats the latter observation as a warning about retrieval quality: the engineer has looked past the existence of the embeddings to question whether they will support the intended behavior.
Product mindedness means accepting that useful applications are still being discovered. If substantial value remains unexplored, today’s product cannot be assumed to be the final form. Bischof hopes teams will look back at their current work in two years and find it naive, because that would mean they have learned enough to build something better.
Urgency matters in any engineering team, but becomes especially valuable when the ground keeps moving. Bischof characterizes the AI landscape as changing underneath builders every three months. The useful trait is the willingness to make progress amid that change; his accompanying ADHD aside is a personal joke, not a candidate requirement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the interview resemble the work
Bischof rejects LeetCode-style interviews for this role because he cannot connect the signal they produce to the work of building AI products. He makes the criticism as someone who says he performs well in those interviews. His proposed replacement tests data and product intuition directly.
His take-home is fundamentally a data-cleaning exercise. Candidates sometimes worry that it was too easy. A good submission, however, has done exactly what he needs: avoided unnecessary complexity, extracted meaning from the data, and drawn conclusions. His coding challenge is similarly approachable by design. Difficulty is not the objective; observing useful engineering behavior is. Hex also includes a product design interview.
Beyond the exercises, look for people who are paying attention to actual products. Enthusiasm is welcome, but trying other AI applications and developing opinions about what works provides more evidence of judgment. One successful hire had written a blog post about AI design patterns. Its treatment of design thinking and how AI should feel gave Bischof a strong indication of her attentiveness before the process was complete. The post did not replace the hiring assessment; she was also technically competent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build with the people whose work the AI supports
AI Leader GPT returns for one more exchange, joking that Bischof has been presenting its ideas as his own. Its bonus advice is the most consequential recommendation beyond hiring: work with domain experts. This relationship should extend beyond asking people to label data or participate in a narrowly defined human-in-the-loop process. Work directly with people who understand the task, and model the AI capability after interactions with them.
For a customer-support bot, that means having customer-support staff use it every day. For Hex’s data science copilot, Bischof brings both his own experience as a data scientist and continuous contact with other practitioners. He says he talks with Hex’s data scientists every week without exception and asks them to use every feature. Their participation gives the team a way to understand whether the capability fits the work.
Engineering skill does not eliminate the need for domain expertise. Bischof’s deliberately narrow exception is an AI product that generates ML and AI products: in that case, the builders are still consulting the relevant experts—themselves. For other domains, access to practitioners remains a central part of building the team around the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inside the take-home and feedback interview
The audience asks how to elicit data intuition rather than simply test coding. Bischof gives candidates a large dataset and asks them to form opinions about it. The task resembles a clustering problem, but there are no objectively good clusters waiting to be recovered. The goal is to extract latent meaning and explain a defensible interpretation.
He uses a take-home because analysis under someone’s continuous observation poorly resembles normal work. Candidates receive seven days to complete the exercise, inspect the data, and write a short report. A live interview follows, centered on their notebook or proposal. That conversation lets Bischof assess the work and correct his own misunderstandings: an approach that looked questionable on first reading may make sense once the candidate explains it.
The review also makes the hiring process a two-way test of working together. Candidates experience the manager’s feedback style before deciding whether to join. The manager sees how they respond to feedback. Excellent work still creates room for discussion about where to go next. Having something useful to say is part of the leader’s responsibility, not something reserved for submissions with obvious flaws.
Scope is another part of the assessment. Bischof allows four hours of work within the seven-day window. An apparently fifteen-hour submission is a red flag; a deliverable well scoped to four hours is a green flag. The exercise tests whether a candidate can identify a minimum useful deliverable, not how much unpaid effort they can pour into an interview.
A small submission can also become a useful conversation. Bischof describes asking one candidate why they had stopped earlier than expected. The candidate saw little value in going deeper before reviewing the initial work together. He hired that person. What initially looked undersized revealed a willingness to seek feedback before spending more effort—another signal the live review made visible.
The product design interview uses a different setting: candidates meet with the design team and design a physical product. Bischof does not describe the detailed prompt, but he recalls taking the interview himself and leaving more eager to join Hex. The assessment let him experience how the company thought about products, just as the feedback review lets candidates experience how a manager works.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where security belongs
An audience member points out the missing security role. Bischof initially places specialist security responsibility outside the team building AI capabilities, with organizational security professionals helping the team make decisions. AI engineers should still be security savvy. Strong software engineers need to anticipate adversarial behavior and recognize the risks their changes introduce.
A participant from Los Alamos National Laboratory challenges that boundary. In a heavily federally regulated environment, they have had success embedding a smart, engaged security professional in the product team from the beginning. This avoids repeatedly bringing an external security team up to speed and trying to attach security late to a defense product.
Bischof accepts the qualification. Different sectors can require a different team structure, and his own environment also handles sensitive customer data, maintains agreements with providers, and places substantial security responsibility on him as team lead. The participant observes that these practices already amount to security being built into the product team. Bischof agrees. The exchange shifts the question from where a security specialist reports to how security participates in the product’s decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Help existing teams through shared AI infrastructure
The next question moves from forming a new team to enabling an existing one, including established engineers who may be reluctant to change how they work. Bischof’s answer focuses on organizational infrastructure: at least one team should be responsible for making AI development easier for the rest of the company.
Individual product teams should not each need to establish a separate relationship with OpenAI, invent prompt infrastructure, and determine how their evaluation system will work. His preferred model centralizes those responsibilities. Product teams consume the resulting capabilities as they would other infrastructure, with one person on each product team responsible for interfacing with the platform team.
His reference point is Stitch Fix, where the data platform team’s charter was to help data scientists move as quickly as possible. As a consumer of that platform, he experienced its value while building and deploying models. He sees a similar opportunity for a centralized AI platform team to support product development.
Bischof acknowledges an organizational alternative, attributing to Netflix a model in which individual product teams can choose to build from scratch. He considers that a fit for Netflix’s circumstances rather than the default for most companies. His answer to reskilling is therefore not a training curriculum: it is a way to reduce the infrastructure each existing team must learn to construct and operate independently.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What can you develop after hiring?
The final question asks which attributes are trainable. Could an enthusiastic software engineer succeed without much data literacy? What about a data scientist who lacks urgency? Bischof wants an existing kernel of each of his three attributes—data intuition, product mindedness, and urgency—that he can help develop. A complete absence of any one worries him.
He then adds the trait he had not named earlier: curiosity. It matters strongly in his hiring judgment, and he says he has never personally succeeded in training it. Skills can grow from a starting point; he wants the desire to investigate and learn to be present when someone joins.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The 2023 essay arguing for an engineering role focused on turning foundation models into useful applications.
Bischof and five fellow practitioners explain LLM development tactics, operational practices, evaluations, and team strategy.
The original paper introducing MMLU, a benchmark measuring language-model accuracy across 57 academic and professional tasks.
Further reading
A March 2024 introduction to Hex's AI assistance for SQL, Python, and analytical workflows.
Full transcript
Loading transcript…