AI Engineer World's Fair 2024
Claude plays Minecraft: Introducing a real world serverless AI to a virtual world
Loading slides…
Read the talk
Claude plays Minecraft: Introducing a real world serverless AI to a virtual world
Derek Bingham’s Minecraft bot Rocky connects conversational requests to game actions, using managed agent orchestration while a stateful client handles the world.
From a talk by Derek Bingham
Before you start: Basic familiarity with LLM tool calling and client–server applications will help; no Minecraft programming experience is required.
Giving Minecraft chat a body
How do you turn a message in Minecraft chat into an agent that actually does something in the game? Derek Bingham begins with the familiar agent loop: accept a request, use an LLM to select tools, run one or more rounds of tool use, and return a response. Minecraft makes that loop visible. A successful request can change where a bot stands or which blocks remain in the ground.
The game already supplies the input interface: its chat function lets players communicate with the world and with one another. It also supplies a rich action space—building, digging, farming, and interacting with animals. Put the agent inside a bot, connect those actions as tools, and a chat message becomes a request for behavior. Both the request and the resulting behavior can be open-ended, which is part of the risk of demonstrating an LLM live.
After one or more orchestration steps, the agent returns a response to Minecraft. Bingham names the bot Rocky, after Bedrock, and gives it tools including jumping, moving to a position, and locating a player. These concrete operations are the bridge between a conversational request and an observable result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a weather question to a hole in the ground
Bingham introduces Rocky as an open-source demonstration that attendees can try with a Minecraft client, then plays a recording made at the AWS booth. It is raining in the game. Asked about the weather, Rocky addresses the player by name, reports the local rain, and suggests taking shelter. The response connects friendly conversational behavior to the state of the world.
The requests then move from observation to action. Rocky jumps on command. Bingham hides behind a building Rocky previously constructed and asks the bot to find him; Rocky announces its approach and comes around to the player. Next, he asks Rocky to find a pig. The Minecraft plugin supplies entity locations, so Rocky can report where a pig is before a subsequent request sends it to that location. Finding an entity and navigating to it are distinct operations in this exchange.
Once Rocky reaches the pig, Bingham asks it to hit the animal, and it does. He notes that repeated requests to hit pigs have been a conspicuous feature of conference demonstrations—human behavior proving at least as interesting as the model’s. He then calls Rocky back and asks for a two-by-two hole. Unlike a bare request to dig, this instruction supplies dimensions that must become action parameters. Rocky digs the hole, then later digs its way out to reach the player. Bingham reports that the team had not expected that escape behavior; the demonstration does not isolate which part of the system produced it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping game state close to the bot
Rocky began as a project for a serverless conference, with three constraints: run in a managed environment, use serverless infrastructure, and make an engaging demonstration. Minecraft and Mineflayer, the bot framework, could run alongside one another in a container. Mineflayer retained the bot’s state, so Bingham kept it out of Lambda. That is an architectural choice about a continuously running game client, rather than a blanket claim that Lambda can retain nothing: AWS documents execution-environment and connection reuse, but reuse is not the same contract as a persistent bot process.
The first orchestration layer used LangChain on Lambda. As the team added actions and tools, Bingham found it increasingly complex, particularly given his limited Python experience. That prototype used Cohere models through Amazon SageMaker. The next version moved orchestration to Agents for Amazon Bedrock and kept Minecraft and Mineflayer on Amazon ECS.
| Version | Minecraft and Mineflayer | Agent and model layer |
|---|---|---|
| Initial prototype | Container | LangChain on Lambda; Cohere through SageMaker |
| Revised hosted design | Amazon ECS | Agents for Amazon Bedrock |
| Onstage demonstration | Bingham’s laptop | Remote Agents for Amazon Bedrock |
The live demonstration uses the same separation between the game client and managed orchestration, but runs the game components locally. Its remote agent calls therefore depend on the venue’s internet connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Managed orchestration, local execution
Amazon Bedrock provides a common API over models from Amazon, Anthropic, Cohere, and other providers, including the Llama family. Rocky’s revised implementation uses Claude, with Bingham choosing the Haiku variant for speed. He gives this as a selection rationale, without reporting a measured latency comparison.
Around that model interface, Bedrock offers agents, knowledge bases for retrieval-augmented generation, guardrails, and evaluations. For Rocky, the relevant managed capabilities include help with prompt creation and orchestration across multiple tasks. Bingham also describes inspecting the agent’s orchestration trace in the console or logs, calling it the agent’s chain of thought. These are the service’s exposed traces, useful for following its actions and responses.
Return control connects the managed agent to the stateful game client. Bingham singles it out as essential to the demonstration. With return control, an action selection comes back to the application for execution. Bedrock can decide what operation to request while the client that maintains the Minecraft connection carries it out.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Defining Rocky’s personality and action parameters
In the software walkthrough, Bingham shows Mineflayer and the project’s source, then opens the Bedrock console. The console exposes model access and the Minecraft agent, but manual configuration is not required: he says the repository includes CloudFormation and CDK definitions. This is the historical Bedrock Agents implementation; the current AWS invocation reference calls the service Agents Classic and says it is closed to new customers, while existing customers may continue using it.
The selected model is Claude 3 Haiku. Its system prompt describes Rocky as a playful, friendly, creative Minecraft agent whose goal is to entertain players and collaborate with them. Bingham explains that the team developed this prompt over time. Personality is configured alongside the task, so Rocky’s chat responses can remain consistent while its tools perform quite different operations.
All the displayed actions use return control. More precisely, the application receives the selected action’s inputs and an invocation ID—not the output of an action already completed in Minecraft. It executes the action and can send the result back through sessionState in another InvokeAgent request. That distinction matters: a request to dig is still pending until the game client performs it.
The dig action exposes depth and width. The earlier two-by-two request supplies those values explicitly; for a small hole, Bingham says the model infers one by one. The boundary between language interpretation and execution can therefore be expressed as a small parameter contract. In JavaScript, a validation helper for those demonstrated dimensions could look like this:
javascript
function readDigDimensions(parameters) {
const values = Object.fromEntries(
parameters.map(({ name, value }) => [name, value])
);
const depth = Number(values.depth);
const width = Number(values.width);
if (!Number.isInteger(depth) || depth < 1 ||
!Number.isInteger(width) || width < 1) {
throw new Error('Dig depth and width must be positive integers');
}
return { depth, width };
}
const requestedHole = readDigDimensions([
{ name: 'depth', value: '2' },
{ name: 'width', value: '2' }
]);
requestedHole holds the requested dimensions; it does not itself dig anything. The model interprets the player’s language, while the client needs usable parameters before it changes the world. Jumping and checking whether it is raining are separate action definitions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Asking for a double-decker couch
A separate action set, Minecraft experimental, contains a single build action. Construction is harder than jumping or reporting weather because the result must occupy a three-dimensional space. The action returns control to the client, where the structure request is used in a construction prompt.
Bingham starts the stopped client, Rocky spawns, and the bot joins the running Minecraft game. He sets the game time to noon so the audience can see, then checks the basic loop by asking Rocky to come to him and dig a small hole. With those actions working, he asks the audience what Rocky should build.
The first suggestion is the Coliseum. Bingham passes it over with a joke about spelling and accepts a double-decker couch instead. He mentions having built a rocket ship the previous day, calls Rocky out of the hole, and gives the bot space. The live request is to build one double-decker couch.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Making a structure description executable
Once the request is sent, Bingham opens the builder prompt. It casts Claude as an expert Minecraft builder created by Anthropic, takes the structure description as input, and asks for valid JSON. The prompt supplies the representation that Rocky’s construction code consumes, giving the model a concrete output shape rather than asking for a prose explanation of how to build a couch.
The prompt also constrains how blocks are placed relative to one another and which block types are allowed. Bingham says that without these rules, the generated structures become nonsense. Valid JSON is only the first constraint; the contents must also describe a buildable arrangement. The structure description determines what to make, while the output format and placement rules determine how that intention reaches the game.
Bingham describes the generated positions as X and Y coordinates, although Minecraft construction is three-dimensional. The Mineflayer API uses x, y, and z positions; the walkthrough does not establish Rocky’s complete JSON schema. Its generated JSON is best understood as input to the application’s construction path, rather than a verified native Mineflayer structure format. The client begins building from that generated arrangement.
Back in Minecraft, Rocky has completed the double-decker couch. Bingham moves to sit on it and emphasizes that the shape is Claude’s interpretation of the request in three-dimensional space. The result is not a fixed couch design selected from a menu: the description has become a constrained representation, then physical blocks in the game. That completed structure makes the whole agent loop visible, from a player’s words to a changed world.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Continuing with Rocky
The demonstration closes with a QR code displayed inside Minecraft. Bingham asks attendees to complete the session feedback form, offering AWS credit codes and a GitHub link to those who do, and invites further questions at the AWS booth. The invitation follows a working example: an agent choosing actions remotely, a client maintaining its connection to the world, and a generated structure made visible one block at a time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
JavaScript framework for creating Minecraft bots, with setup instructions and examples.
Explains how an application receives an agent's action request, executes it, and returns the result.
Further reading
Documents bot actions, three-dimensional positions, digging, and block placement.
Updates since the talk
- InvokeAgent API and current availabilityDocumentation
Current invocation reference, including a notice that Bedrock Agents Classic is no longer open to new customers.
Full transcript
Loading transcript…