AI Engineer World's Fair 2024

Claude plays Minecraft: Introducing a real world serverless AI to a virtual world

Loading slides…

Read the talk

Claude plays Minecraft: Introducing a real world serverless AI to a virtual world

Derek Bingham’s Minecraft bot Rocky connects conversational requests to game actions, using managed agent orchestration while a stateful client handles the world.

From a talk by Derek Bingham

Before you start: Basic familiarity with LLM tool calling and client–server applications will help; no Minecraft programming experience is required.

Giving Minecraft chat a body

How do you turn a message in Minecraft chat into an agent that actually does something in the game? Derek Bingham begins with the familiar agent loop: accept a request, use an LLM to select tools, run one or more rounds of tool use, and return a response. Minecraft makes that loop visible. A successful request can change where a bot stands or which blocks remain in the ground.

The game already supplies the input interface: its chat function lets players communicate with the world and with one another. It also supplies a rich action space—building, digging, farming, and interacting with animals. Put the agent inside a bot, connect those actions as tools, and a chat message becomes a request for behavior. Both the request and the resulting behavior can be open-ended, which is part of the risk of demonstrating an LLM live.

An Agents slide connects users to a Minecraft grass block, a green bot and a pickaxe, with an LLM above the bot and a loop below it.Open full source frame
The agent workflow mapped to Minecraft, with a bot, an LLM and a pickaxe.

After one or more orchestration steps, the agent returns a response to Minecraft. Bingham names the bot Rocky, after Bedrock, and gives it tools including jumping, moving to a position, and locating a player. These concrete operations are the bridge between a conversational request and an observable result.

0:000:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From a weather question to a hole in the ground

Bingham introduces Rocky as an open-source demonstration that attendees can try with a Minecraft client, then plays a recording made at the AWS booth. It is raining in the game. Asked about the weather, Rocky addresses the player by name, reports the local rain, and suggests taking shelter. The response connects friendly conversational behavior to the state of the world.

Rocky stands beside a wooden building in the rain. Chat asks about the weather, and Rocky replies that it is rainy and suggests shelter.Open full source frame
Rocky answers a weather question while rain falls in Minecraft.

The requests then move from observation to action. Rocky jumps on command. Bingham hides behind a building Rocky previously constructed and asks the bot to find him; Rocky announces its approach and comes around to the player. Next, he asks Rocky to find a pig. The Minecraft plugin supplies entity locations, so Rocky can report where a pig is before a subsequent request sends it to that location. Finding an entity and navigating to it are distinct operations in this exchange.

Once Rocky reaches the pig, Bingham asks it to hit the animal, and it does. He notes that repeated requests to hit pigs have been a conspicuous feature of conference demonstrations—human behavior proving at least as interesting as the model’s. He then calls Rocky back and asks for a two-by-two hole. Unlike a bare request to dig, this instruction supplies dimensions that must become action parameters. Rocky digs the hole, then later digs its way out to reach the player. Bingham reports that the team had not expected that escape behavior; the demonstration does not isolate which part of the system produced it.

3:283:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:28 · section reference included

Keeping game state close to the bot

Rocky began as a project for a serverless conference, with three constraints: run in a managed environment, use serverless infrastructure, and make an engaging demonstration. Minecraft and Mineflayer, the bot framework, could run alongside one another in a container. Mineflayer retained the bot’s state, so Bingham kept it out of Lambda. That is an architectural choice about a continuously running game client, rather than a blanket claim that Lambda can retain nothing: AWS documents execution-environment and connection reuse, but reuse is not the same contract as a persistent bot process.

The first orchestration layer used LangChain on Lambda. As the team added actions and tools, Bingham found it increasingly complex, particularly given his limited Python experience. That prototype used Cohere models through Amazon SageMaker. The next version moved orchestration to Agents for Amazon Bedrock and kept Minecraft and Mineflayer on Amazon ECS.

VersionMinecraft and MineflayerAgent and model layer
Initial prototypeContainerLangChain on Lambda; Cohere through SageMaker
Revised hosted designAmazon ECSAgents for Amazon Bedrock
Onstage demonstrationBingham’s laptopRemote Agents for Amazon Bedrock

The live demonstration uses the same separation between the game client and managed orchestration, but runs the game components locally. Its remote agent calls therefore depend on the venue’s internet connection.

6:216:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:21 · section reference included

Managed orchestration, local execution

Amazon Bedrock provides a common API over models from Amazon, Anthropic, Cohere, and other providers, including the Llama family. Rocky’s revised implementation uses Claude, with Bingham choosing the Haiku variant for speed. He gives this as a selection rationale, without reporting a measured latency comparison.

Around that model interface, Bedrock offers agents, knowledge bases for retrieval-augmented generation, guardrails, and evaluations. For Rocky, the relevant managed capabilities include help with prompt creation and orchestration across multiple tasks. Bingham also describes inspecting the agent’s orchestration trace in the console or logs, calling it the agent’s chain of thought. These are the service’s exposed traces, useful for following its actions and responses.

Return control connects the managed agent to the stateful game client. Bingham singles it out as essential to the demonstration. With return control, an action selection comes back to the application for execution. Bedrock can decide what operation to request while the client that maintains the Minecraft connection carries it out.

8:188:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:18 · section reference included

Defining Rocky’s personality and action parameters

In the software walkthrough, Bingham shows Mineflayer and the project’s source, then opens the Bedrock console. The console exposes model access and the Minecraft agent, but manual configuration is not required: he says the repository includes CloudFormation and CDK definitions. This is the historical Bedrock Agents implementation; the current AWS invocation reference calls the service Agents Classic and says it is closed to new customers, while existing customers may continue using it.

The selected model is Claude 3 Haiku. Its system prompt describes Rocky as a playful, friendly, creative Minecraft agent whose goal is to entertain players and collaborate with them. Bingham explains that the team developed this prompt over time. Personality is configured alongside the task, so Rocky’s chat responses can remain consistent while its tools perform quite different operations.

Amazon Bedrock console with Anthropic Claude 3 Haiku selected, instructions describing Rocky as a playful Minecraft agent, an Action groups section and a Test Agent panel.Open full source frame
Rocky's configuration shows Claude 3 Haiku and its Minecraft agent instructions.

All the displayed actions use return control. More precisely, the application receives the selected action’s inputs and an invocation ID—not the output of an action already completed in Minecraft. It executes the action and can send the result back through sessionState in another InvokeAgent request. That distinction matters: a request to dig is still pending until the game client performs it.

The dig action exposes depth and width. The earlier two-by-two request supplies those values explicitly; for a small hole, Bingham says the model infers one by one. The boundary between language interpretation and execution can therefore be expressed as a small parameter contract. In JavaScript, a validation helper for those demonstrated dimensions could look like this:

javascript

function readDigDimensions(parameters) {
  const values = Object.fromEntries(
    parameters.map(({ name, value }) => [name, value])
  );
  const depth = Number(values.depth);
  const width = Number(values.width);

  if (!Number.isInteger(depth) || depth < 1 ||
      !Number.isInteger(width) || width < 1) {
    throw new Error('Dig depth and width must be positive integers');
  }

  return { depth, width };
}

const requestedHole = readDigDimensions([
  { name: 'depth', value: '2' },
  { name: 'width', value: '2' }
]);

requestedHole holds the requested dimensions; it does not itself dig anything. The model interprets the player’s language, while the client needs usable parameters before it changes the world. Jumping and checking whether it is raining are separate action definitions.

11:0911:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:09 · section reference included

Asking for a double-decker couch

A separate action set, Minecraft experimental, contains a single build action. Construction is harder than jumping or reporting weather because the result must occupy a three-dimensional space. The action returns control to the client, where the structure request is used in a construction prompt.

Bingham starts the stopped client, Rocky spawns, and the bot joins the running Minecraft game. He sets the game time to noon so the audience can see, then checks the basic loop by asking Rocky to come to him and dig a small hole. With those actions working, he asks the audience what Rocky should build.

The first suggestion is the Coliseum. Bingham passes it over with a joke about spelling and accepts a double-decker couch instead. He mentions having built a rocket ship the previous day, calls Rocky out of the hole, and gives the bot space. The live request is to build one double-decker couch.

13:1713:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:17 · section reference included

Making a structure description executable

Once the request is sent, Bingham opens the builder prompt. It casts Claude as an expert Minecraft builder created by Anthropic, takes the structure description as input, and asks for valid JSON. The prompt supplies the representation that Rocky’s construction code consumes, giving the model a concrete output shape rather than asking for a prose explanation of how to build a couch.

The prompt also constrains how blocks are placed relative to one another and which block types are allowed. Bingham says that without these rules, the generated structures become nonsense. Valid JSON is only the first constraint; the contents must also describe a buildable arrangement. The structure description determines what to make, while the output format and placement rules determine how that intention reaches the game.

Bingham describes the generated positions as X and Y coordinates, although Minecraft construction is three-dimensional. The Mineflayer API uses x, y, and z positions; the walkthrough does not establish Rocky’s complete JSON schema. Its generated JSON is best understood as input to the application’s construction path, rather than a verified native Mineflayer structure format. The client begins building from that generated arrangement.

Back in Minecraft, Rocky has completed the double-decker couch. Bingham moves to sit on it and emphasizes that the shape is Claude’s interpretation of the request in three-dimensional space. The result is not a fixed couch design selected from a menu: the description has become a constrained representation, then physical blocks in the game. That completed structure makes the whole agent loop visible, from a player’s words to a changed world.

16:0016:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:00 · section reference included

Continuing with Rocky

The demonstration closes with a QR code displayed inside Minecraft. Bingham asks attendees to complete the session feedback form, offering AWS credit codes and a GitHub link to those who do, and invites further questions at the AWS booth. The invitation follows a working example: an agent choosing actions remotely, a client maintaining its connection to the world, and a generated structure made visible one block at a time.

A large QR code on a pale square stands above a grassy Minecraft landscape, with the speaker inset at lower left.Open full source frame
A QR code displayed inside Minecraft closes the demonstration.
17:2117:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:21 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…