AI Engineer Summit 2023
Using AI to Build an Infinite Game
Loading slides…
Read the talk
Using AI to Build an Infinite Game
Jeff Schomay turns a finite forest adventure into a stream of generated encounters by combining scene JSON, a custom image model, validation, and a replenished asset cache.
From a talk by Jeff Schomay
Before you start: Familiarity with JSON and basic API request flows will help; no game-development or model-training experience is required.
A forest the player can exhaust
You are lost in a forest. Each encounter changes your vigor or courage, and you need to find home before your courage runs out. In Jeff Schomay’s Lost in an Infinite Forest, the original content was entirely AI-generated, but the world was still finite: sixteen scenes arranged in a four-by-four grid. After a few playthroughs, a player could see everything.
The most enjoyable part of building that world had been generating a scene and discovering what the AI would produce. Why reserve that surprise for the developer? If new scenes could be generated for the player, each game could offer different encounters. Infinite exploration begins by moving generation from content creation into the game’s asset supply.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate a scene the game can consume
The first requirement is a consistent scene definition. A scene is a JSON object describing what happens when the player discovers it, what happens on a return visit, and how those encounters affect player stats. That distinction gives a location behavior beyond its first appearance: returning to a scene can produce different text and consequences. The visible “Intimidating Bear” example includes discovery and return descriptions with courage changes.
Schomay starts with OpenAI’s completion endpoint and a long, detailed prompt. Most of the time, it produces the expected JSON structure and content that fits the forest while remaining varied and interesting. He then tries fine-tuning to make that output more reliable. This is the historical completion-and-fine-tuning workflow; the current OpenAI guide says its fine-tuning platform is winding down and unavailable to new users.
Schomay generates fifty examples for fine-tuning, citing OpenAI’s recommendation at the time of fifty to a hundred. He shortens the training prompt, replacing the detailed formatting instructions with a general description of the desired scene so the examples carry more of the output contract. His spoken account says he removes all mention of JSON; the published asset-server example still requests a specific JSON map. The supported distinction is between supplying the full structure in the prompt and teaching that structure through examples.
Schomay reports spending roughly $1–$2 on generating the examples and fine-tuning combined. In his trial, he reports perfect JSON output even without mentioning JSON in the prompt, although he gives no evaluation sample size or measured failure rate. The shorter prompt also uses fewer tokens, which he describes as faster, cheaper, and easier to work with. Fine-tuning has shifted repeated instructions into the model’s learned behavior; the later asset pipeline will still validate its output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the style consistent and the encounters varied
Next comes the artwork. Leonardo offers both image generation and custom image models, allowing Schomay to pursue a consistent visual style across the game. He experiments with prompts and generation parameters until he finds a look he likes. Surprisingly, the scene description itself works as the image prompt, despite containing second-person narration and details beyond visible objects. That lets the text generator’s output feed directly into image generation.
Training the image model requires separating what should stay consistent from what should vary. If every training image repeats the same subjects, the model can overfit and make every scene look alike. If the images share too little, the desired style becomes unclear.
| Keep consistent | Allow to vary |
|---|---|
| Perspective and scale | People and animals |
| Forest setting | Buildings |
| Overall tone and texture | Scenes without those subjects |
This is more demanding than collecting attractive forest pictures: the training set has to demonstrate both the visual rules and the range of encounters those rules should accommodate. Schomay tries multiple models with different parameters and image sets before finding a satisfactory result.
He tests the chosen model by generating a large collection of images. Recurring trees, a zigzag path down the middle, and a shared surface treatment make the scenes feel related, while their contents remain distinct. With structured encounters and cohesive artwork in place, the next task is to connect them into a working asset pipeline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validate, illustrate, and assemble
The asset server joins the two generators in a dependent sequence:
- Request a scene from OpenAI using the custom model.
- Validate the returned JSON, checking that the required keys are present.
- Send the valid scene’s description to Leonardo’s custom image model.
- Combine the returned image with the scene data, then save and deliver the asset.
Validation sits before image generation, so a scene missing required fields does not proceed to the next generation step. The completed diagram shows the whole path, ending with “Merge and save.”
The same control flow can be expressed in Python with provider calls supplied as functions. Here, required_keys and description_key come from the game’s scene format:
python
import json
def build_asset(request_scene, generate_image, save_asset,
required_keys, description_key):
scene = json.loads(request_scene())
if not isinstance(scene, dict):
raise ValueError("Scene must be a JSON object")
missing = set(required_keys) - scene.keys()
if missing:
raise ValueError(f"Missing scene keys: {sorted(missing)}")
description = scene[description_key]
if not isinstance(description, str) or not description.strip():
raise ValueError("Scene description must be nonempty text")
image = generate_image(description)
asset = {"scene": scene, "image": image}
save_asset(asset)
return asset
The scene description is the connection between narrative generation and artwork; the final asset keeps the image and the encounter data together.
Schomay also builds a simple preview server that lets him scroll through generated scenes. The pipeline’s structural checks establish that data has the necessary fields; previewing lets him inspect the assembled text and artwork before integrating the output into play.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare encounters before the player needs them
The game can now request generated assets when the player enters a new scene, but that exposes the pipeline’s waiting time. Schomay reports scene-and-image generation taking 10, 20, sometimes 30 seconds. Waiting that long at each transition would interrupt exploration.
His solution is a prefilled scene cache. Generate a stock of complete scenes before they are needed, take one when the player enters a new location, and replenish the stock whenever it falls below a threshold. The player receives a prepared encounter while generation supplies future encounters. This does not make either model faster; it moves their work ahead of consumption. Keeping scenes ready depends on replenishment keeping pace with play.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Walk through the generated forest
Schomay introduces the live demonstration as content that has never been seen before and will never be seen again. The game still has a fixed starting point: a lamppost from which the player sets out to find home. Stats appear in the bottom-left corner, but their effects also change how the world feels. Lower vigor slows movement; lower courage makes the viewport smaller. An encounter therefore affects both the player’s resources and their ability to navigate.
Moving down reveals the first generated encounter: soft blue, pulsating light coming from organic formations scattered around a glade. The text describes fear and tiredness lifting. It offers a vigor increase, but the player is already at full vigor. The next location resembles a campfire scene; continuing down leads to a large, dark cave at the end of a path. That encounter is daunting, and courage falls.
The player changes direction and enters foggy trees where visibility is poor, then backtracks through a winding road. Another move brings Schomay back to where he started. The demonstration stops there rather than showing a win: exploration would continue until the player found home, and another playthrough would provide different encounters. Fresh content is being supplied to a navigable world in which the player can still revisit locations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Higher resolution and a more personal forest
The working pipeline leaves room for additional processing. Schomay describes the existing images as 512 pixels and proposes adding an AI upscaler to increase their resolution. That would also add generation time: improved artwork would make each prepared asset more expensive in time to produce.
He also proposes making the scene prompt responsive to the player’s context:
- Selected theme: Let the player choose the kind of forest experience to generate.
- Time of day: Incorporate the player’s current time into the scene request.
- Local weather: Use current weather at the player’s location to influence the generated world.
These are proposed extensions, with the aim of making scenes feel more closely connected to where the player is. The same generation process could also supply assets for other projects: the forest is one application of a pipeline that turns a scene request into structured content and matching artwork.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The forest exploration game, with macOS and Python source downloads and instructions for navigating while managing courage and vigor.
Backend repository covering scene fine-tuning, image generation, asset previewing and a replenished scene cache.
Python game client with installation and executable build instructions.
Further reading
The speaker's written walkthrough of scene prompts, custom artwork and the asset pipeline, with links to the project.
Updates since the talk
API reference for training a custom image model from a dataset, including prompt and resolution parameters.
- OpenAI supervised fine-tuning guidanceDocumentation
Dataset sizing and evaluation guidance, with a current notice about the fine-tuning platform winding down.
Full transcript
Loading transcript…