AI Engineer World's Fair 2024
From Software Developer to AI Engineer
Loading slides…
Read the talk
From Software Developer to AI Engineer
Build on software engineering skills by learning model behavior, using AI development tools, testing models through a common API, and composing tools into agent workflows.
From a talk by Antje Barth and Mike Chambers
Before you start: Familiarity with application development, APIs, and basic Python will help with the examples; no machine learning research background is required.
Learn what sits behind the API
If an AI application can begin with a few API calls, what does a software developer still need to learn? The opening role diagram places AI engineering on the application side of the API boundary. The role associated with swyx’s The Rise of the AI Engineer makes room for developers who build with models without becoming ML researchers or data scientists first. Access to a model removes a substantial barrier to getting started.
An accessible API does not remove the need to understand model behavior. You need enough foundation-model knowledge to understand why an output might differ from what your application expects. You also need to recognize when to customize a model for a particular use case or dataset, including through fine-tuning, and how application functions can give it access to additional systems. Those are separate engineering decisions: understanding the model, adapting it, and connecting it to software.
For that foundation, Antje Barth recommends Generative AI with Large Language Models, a course developed with colleagues, Andrew Ng, and DeepLearning.AI. It introduces generative AI fundamentals with the goal of building real applications and is offered through DeepLearning.AI and Coursera.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use AI across the development lifecycle
The second step is to use AI developer tools in everyday work. Natural language becomes another interface to development: a request can produce a rewritten README, while existing code can become an English explanation or function documentation. The useful direction runs both ways—from a description toward a software artifact, and from code toward understanding.
That expands the opportunity beyond typing code faster. Planning, design, testing, deployment, and maintenance surround implementation. Barth offers a rough workload framing: sometimes up to 70% goes to boilerplate, documentation, and maintenance, leaving perhaps 30% for creative development. This is an illustrative split, not a measured productivity result for Amazon Q; the point is that assistance can target the work around feature implementation too.
Amazon Q Developer is introduced as an assistant for that broader lifecycle. Its agents can tackle feature development and code transformation, in addition to code completion. Moving an older Java codebase to a newer Java version is one proposed transformation use case; the demonstration that follows instead builds up from IDE assistance to a feature change spanning project files.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move from IDE guidance to a reviewed feature change
Mike Chambers begins inside the IDE with an ordinary chat question about creating a serverless application. Q responds with a sequence starting at AWS SAM CLI installation and continuing through project creation. Once generated project code is available, he selects it, opens the context menu, and asks Q to explain it. The selected code becomes part of the explanation prompt. The same interaction can help with an unfamiliar legacy system, where the original author is no longer available to explain the implementation.
Next, Chambers supplies a comment describing an input-checking function, then narrows the requirement: trim any incoming string. Q generates the small function from that local description. This is the familiar completion workflow: the developer states a small intent close to the code, and the assistant supplies an implementation.
The demonstration then grows from a function into a feature. Using the recording’s /dev interface, Chambers requests a DynamoDB search by category and specifies how its output should be formatted. The workflow proceeds through distinct stages:
- Describe the feature. Provide the search requirement and output details in chat.
- Inspect the plan. Q identifies project changes, including work in
template.yamland a proposedsearchByCategory.mjsfile. - Review the change set. Q follows the plan to generate proposed code, exposing differences from the current project.
- Accept the changes. The demonstrated
Insert codebutton applies the generated code to the project if the developer approves it.
A generated plan becomes useful when it leads to inspectable changes. The review step separates a proposal from code accepted into the project; this is more than a longer autocomplete suggestion.
For further examples, Barth points to Amazon Q Developer Center, along with the conference’s Q sessions and AWS booth demonstrations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prototype around a use case, then evaluate models
The third step shifts from using an AI assistant to building an AI application. Start by defining the use case, then choose candidate models and decide how to adapt them. Prompt engineering, retrieval-augmented generation (RAG), and fine-tuning are options to consider against that use case, rather than mandatory stages every application must pass through.
Responsible AI policies, privacy, security, and guardrails belong throughout this workflow. After integration, agents and deployed models also need ongoing maintenance—the operational work often described as GenAI Ops or LLM Ops. Model selection sits within this larger lifecycle: evaluate candidates thoroughly because different application tasks may call for different models. A common application platform does not imply a single best model for every use case.
Amazon Bedrock is presented as the managed platform for this experimentation. It combines access to foundation models with tooling for fine-tuning, RAG workflows, and agents. Barth emphasizes a secure environment and control over application data. The talk’s model lineup includes providers AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, and Stability AI, alongside Amazon’s Titan models.
The release context is June 2024: Barth highlights Claude 3.5 Sonnet as having arrived on Bedrock the previous week. That historical lineup sets up the next integration problem: how to experiment across providers without rewriting the invocation code for each one.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep model invocation consistent with Converse
The Converse API, newly introduced at the time of the recording, provides a common invocation structure. Instead of making the application translate parameter names and message formats for each model, Bedrock handles that translation. The interface accommodates system instructions and user/assistant messages and returns a consistent output structure.
The Python example follows a short sequence: initialize the SDK client, construct messages, and call converse with a model ID and inference parameters. The following uses that same structure, with the model ID supplied through the environment:
python
import os
import boto3
client = boto3.client("bedrock-runtime")
messages = [
{
"role": "user",
"content": [{"text": "Explain how an API helps developers use AI models."}],
}
]
response = client.converse(
modelId=os.environ["BEDROCK_MODEL_ID"],
messages=messages,
inferenceConfig={
"maxTokens": 256,
"temperature": 0.2,
},
)
for block in response["output"]["message"]["content"]:
if "text" in block:
print(block["text"])
The request separates the conversation in messages from generation settings in inferenceConfig. In the recording, the selected model is from Anthropic; the common call structure is what lets the developer experiment with other supported models by changing the model selection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Declare the functions a model can request
Converse also supports function calling for models that provide that capability. The setup described here is to define a list of tools representing application functions, then pass that list with the Converse call. A common invocation interface does not make tool support universal across models. The tool list describes the functions available for use; the next demonstration makes that connection concrete with code-backed actions in an agent.
For more detailed examples across languages, Barth points to the generative AI space on community.aws and names Dennis Traub as a tutorial author.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give an agent instructions and code-backed actions
The fourth step is application integration, illustrated through a Minecraft agent. Chambers opens an agent configured in the AWS console, describing the agent workflow as fully serverless. In the agent builder, the selected model is Claude 3 Haiku, and the instructions tell the Minecraft bot how to play. Those instructions are operational prompts, not notes for the developer.
The agent also needs actions it can perform. Chambers has organized these into Minecraft actions and an experimental action group. Inside the actions group, tools such as jump and dig connect to code. Their descriptions explain their intended use, making the tool descriptions another part of prompt engineering. Required parameters give the agent information it must obtain before using an action.
Two more actions prepare the central demonstration:
| Action | Role in the workflow |
|---|---|
| Get a player’s location | Obtain the destination associated with a player |
| Move to a location | Move the bot toward that destination |
These are small, reusable capabilities. Together, they let the agent address a request whose wording does not correspond to a dedicated action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose actions to answer “Come to me”
Inside Minecraft, Chambers first changes nighttime to daytime so the movement will be visible. Rocky, the Bedrock-powered bot, stands in the game world and can receive messages through chat. Chambers backs away to make the request more interesting. There is no dedicated tool for coming to the player.
He types “Come to me.” The described workflow connects the chat request to the available actions:
- Identify which player sent the request.
- Use that player’s name to obtain their location.
- Use the destination to move Rocky from its current position toward the player.
Chambers describes the final step as mapping a path to him. Rocky reaches the player, and the game chat confirms arrival. The useful result is composition of existing tools into a requested behavior, without adding a special-purpose come-to-player action.
Barth points viewers to further demonstrations at the AWS booth and in the agents track, and says the project code is available on GitHub for people who want to explore the integration themselves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep learning with other builders
The fifth step is continued learning through community participation. As models, tools, and application patterns change, workshops and conversations with other builders become part of staying current. Barth closes by announcing plans for an AWS AI engineering hub in San Francisco, with workshops, events, and meetups.
The invitation includes shaping that program: suggest topics through a survey, volunteer to speak, or offer to host a meetup. At the time of the talk, an Anthropic happy hour was near or at capacity, and additional events were planned for the following weeks and months. The closing opportunity is to participate in the learning community as a contributor as well as an attendee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- The Rise of the AI EngineerArticle
Swyx’s essay introducing the AI engineer role and application development around foundation models.
The May 2024 announcement explains unified model invocation, conversation history and supported function calling.
AWS’s June 20, 2024 announcement documents the model release discussed in the talk.
Full transcript
Loading transcript…