AI Engineer World's Fair 2024

Moondream: how does a tiny vision model slap so hard?

Loading slides…

Read the talk

Moondream: how does a tiny vision model slap so hard?

A compact vision model becomes useful through a narrow task boundary, carefully generated training data, and an interface that lets developers ask for what they need.

From a talk by Vikhyat Korrapati

Questions, captions, coordinates and counts

Where is the parrot’s beak? Locating an object is one of the practical jobs for Moondream, alongside answering questions, captioning images and counting. Its creator, Vikhyat Korrapati, began working on the project after roughly nine years at AWS. The version presented here has fewer than two billion parameters and is available under Apache 2.0.

The opening examples show how these tasks require different outputs: a description for captioning, an answer for a visual question, or coordinates for localization. In the object-detection frame, a green bounding box encloses the parrot’s beak.

A parrot with a green rectangle around its beak, beside an Object Detection panel labeled “Bounding box: beak.”Open full source frame
Moondream’s object-detection example marks a parrot’s beak with a green bounding box.

Across VQA v2 and GQA, Korrapati shows improvements over three months of releases and reports performance comparable to LLaVA 1.5, a model with seven billion parameters. That comparison establishes the opportunity: a much smaller model can be competitive on these visual-question-answering tasks. It does not establish equivalent capability across everything the larger model can do.

0:000:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Browser automation makes model control a requirement

Moondream began inside a different product idea: an agent for automated browser testing. The agent needed to see the user’s screen, describe the page and answer questions about it. Korrapati initially tried GPT-4V, but his early experience included refusals when images contained people. Anticipated latency and cost created further obstacles. For this application, control over the model became a product requirement.

The task was constrained enough to make building a small model plausible. Describing screens and answering questions about them did not require general intelligence. With a couple of RTX 3090s at home, Korrapati planned to train a small prototype, then rent larger cloud machines for a bigger version.

The small version worked better than he expected. He posted it on Twitter expecting modest interest and a quick return to the application. Instead, the response was large enough to change the project’s direction.

Automated-testing companies began asking whether they could use the model to describe browser screens. Other developers were interested too; Korrapati gives Open Interpreter a shout-out. He paused the testing application to concentrate on Moondream and find out how far a reusable vision component could go.

1:502:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

Define the perception task, then choose the components

The task boundary determines the data and the evaluation. Moondream is intended as a developer tool for accurate image understanding with fewer hallucinations. Broad world knowledge and creative writing are outside that focus. Asking it to write a poem spends capacity on something the product does not need. Those exclusions make it easier to decide which training examples and benchmarks deserve attention.

Mathematical reasoning provides a concrete boundary. Solving problems in a benchmark such as MathVista is not the goal. For Korrapati’s example of a photographed differential equation, producing a LaTeX representation could be useful: it extracts what is visible. Solving the calculus problem is a separate capability that Moondream does not aim to acquire.

“Define your niche” slide with four bullets covering accuracy, visual understanding, data and benchmark choices, and excluding calculus.Open full source frame
Moondream’s niche emphasizes accuracy, reduced hallucinations, and seeing rather than solving calculus.

The model also starts from existing components. It combines Google’s SigLIP vision encoder with Microsoft’s pretrained Phi-1.5 language model. Korrapati describes Phi-1.5’s largely synthetic training data as a good fit for Moondream’s own pipeline.

For this task and budget, he found training from scratch cost prohibitive without a meaningful benefit. Experiments with other pretrained models also made relatively little difference. Training data was the larger source of improvement, so that became the focus of the work.

3:373:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:37 · section reference included

Synthetic supervision must fit the smaller model

Korrapati reports that the latest release discussed was trained on around 35 million images. Obtaining useful supervision at that scale creates two immediate problems:

  • Human annotation: High-quality annotation is expensive. He relays a rumor that some providers require seven-figure upfront commitments before engaging.
  • Internet image–alt-text pairs: These are plentiful, but often noisy and formatted differently from the questions and answers the model needs to learn. Korrapati finds that noise particularly damaging when training small models.

Synthetic processing offers a way to reshape existing information into suitable training examples. But a powerful generator is not automatically a good teacher. Korrapati raises terms-of-use concerns about training on GPT-4 outputs, then identifies a technical problem: those outputs can depend on reasoning and world knowledge that the smaller model cannot reproduce.

His warning is that the student may learn to imitate the teacher’s confident language without learning the basis for its answers. The resulting response sounds plausible but includes unsupported details. The useful target is grounded supervision the small model can learn, not simply the most elaborate answer a larger model can produce.

5:295:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:29 · section reference included

A harbor caption shows two ways synthesis goes wrong

The COCO example makes the problem visible. He describes COCO as roughly 200,000 images, with five short descriptions per image and object annotations that include coordinates. The desired transformation sounds straightforward: combine those short descriptions and annotations into a detailed caption containing their collective information.

Naively supplying that material to GPT-4 produces a fluent harbor description, but fluency conceals two different errors.

Error sourceWhat happens in the harbor caption
Unreliable annotationA tiny feature that might be a person or a post becomes a definite person near the right side of the harbor.
Generated embellishmentThe caption says that five yachts stand out from the rest.

The first error turns ambiguous source information into certainty. The second adds an interpretation that the supplied information does not justify. Combining annotations into longer prose therefore needs more than a request for detail: the process must control which facts survive and what the generator is allowed to add.

6:597:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:59 · section reference included

Turn spatial narratives into grounded questions

Preprocessing becomes central in the next example, Google’s Localized Narratives dataset. Annotators describe an image aloud while moving the mouse over the region they are describing. This encourages detailed descriptions tied to spatial positions. A description of the girl in front playing a guitar carries both an activity and a relationship within the scene—useful supervision for spatial reasoning, which vision-language models often struggle with.

The processing slide lays out the stages used to turn that material into training questions:

  1. Correct transcription errors.
  2. Extract facts.
  3. Validate those facts.
  4. Create question–answer pairs.
  5. Filter unhelpful pairs.
  6. Generate absurd or irrelevant questions.

Fact extraction and validation give the pipeline explicit opportunities to clean the information before it becomes a question and answer. Filtering then removes unhelpful results, while the final stage introduces questions the image cannot support.

The pipeline used about 20 calls to Mixtral 8x7B per image. Korrapati estimates that generating training data consumed one or two orders of magnitude more compute than training Moondream itself. In his experience, that expense was justified because training-data quality moved model performance more than the other changes he tried.

He released the resulting dataset on Hugging Face. One generated example asks, “What theory does the kid have about the existence of pleasure in the image?” A photograph cannot establish that philosophical belief. These distractor questions teach an essential behavior: the model should not accept a question’s premise merely because the user supplied it.

Processing steps: correct transcription errors, extract facts, validate facts, create question–answer pairs, filter unhelpful pairs, and generate absurd or irrelevant questions.Open full source frame
A six-step Mixtral 8x7B pipeline converts Localized Narratives into question–answer pairs and generates distractors.
7:578:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:57 · section reference included

Match real queries and break synthetic shortcuts

Even a simple formatting mismatch can expose a gap in synthetic training data. An early user submitted a straightforward question that Moondream could not answer. Korrapati traced the failure to its lowercase first letter: the training questions had not included that variation.

The fix was to add capitalization changes and typos to the training data. The broader requirement is distribution matching. Training inputs need to resemble the queries people actually submit, including their imperfections.

The generator’s own habits create another failure mode, which Korrapati discusses under model collapse. When asked to invent questions irrelevant to an image, Mixtral repeatedly produced questions about dinosaurs and aliens. Training on that pattern risks teaching the student a topic shortcut: reject questions about dinosaurs and aliens. That is different from learning whether the image supports the question.

The generation process therefore needs variation beyond the generator’s habitual choices. For synthetic captions, one option is to provide the image’s alt text as additional context. The instruction must allow that text to be noisy or irrelevant and use only facts that are relevant. This introduces outside information without requiring the model to trust all of it.

9:459:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:45 · section reference included

Community feedback and the option to self-host

The community helped establish that a reusable vision model might be more valuable than the original testing application. Other developers wanted to build their own applications around the same capability. Direct conversations on Twitter gave Korrapati a short path from discovering a need to asking a potential user about it. The community also brought partners, mentors and practical support.

Free competing tools influenced the decision to release Moondream as open source, but the resulting benefits went beyond price:

  • Developer portability: Users could run the model in the environments their applications required.
  • Enterprise independence: An organization could prefer managed operation while retaining the option to run the software itself if a vendor disappeared or changed the terms of the relationship.
  • Community contributions: Contributors helped port the model to additional platforms, including execution in a web browser.

The ability to self-host therefore matters even to customers who do not initially intend to operate the model themselves.

11:0011:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:00 · section reference included

Let the application choose its guardrails

For developer tools, Korrapati favors placing safety guardrails at the application layer. His browser-testing experience explains why: refusing to process an image because it contains a person blocked a legitimate task the application needed to perform. A restriction appropriate to one product can prevent another product from working.

In his view, developers building on a model should be able to choose and implement safeguards suited to their applications. He distinguishes this infrastructure role from a consumer assistant, where building safeguards directly into the model can make sense. The position is about where control belongs in the product stack, not whether safety matters.

12:3612:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:36 · section reference included

Deploy small models close to the decision

Vision applications make efficiency especially consequential because a video stream creates a continuing inference workload. To illustrate how continuous video processing accumulates cost, Korrapati uses 30 frames per second at a stated seven-tenths of a cent per second. This is a cost illustration, not a Moondream serving-price quote. Repeated processing can quickly consume the margin available to an application.

Privacy and latency create separate reasons to run inference close to where decisions happen. Korrapati expects smaller models to become the deployed artifacts for many such applications, while larger models remain useful in development—for example, generating the smaller model’s training data. That division gives the larger model a valuable role without requiring every production inference to use it.

13:3313:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:33 · section reference included

Prompting makes familiar vision tasks easier to adopt

Some users built applications that depended on understanding language and images together. Others used Moondream for familiar computer-vision requests: determine whether a person is present, ask whether something suspicious is happening, or locate a bus in a road-camera image. The demand was not limited to entirely new capabilities.

Established models such as YOLOv3 could already address tasks such as object detection. The change was the amount of work a developer had to take on before trying the idea.

ApproachWork required to specify the task
Train a custom vision modelLearn the workflow, collect and annotate data, then train a model.
Use a promptable vision-language modelDescribe in English what to extract from the image.

Korrapati describes a developer weighing an illustrative two weeks of learning against the value of adding a vision feature. An inexpensive prompt can make that same feature worth trying. Prompting changes the developer experience, even when the underlying task was already technically possible.

14:2714:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:27 · section reference included

Reduce image tokens and expand the team

The roadmap keeps the same product boundary: help developers build vision applications, rather than pursue general intelligence. At the time of the talk, Moondream used 729 tokens to represent an image, which Korrapati links to practical use with only one image at a time.

The planned improvement was an option for a more compressed image representation. The useful tradeoff depends on the task. Reading text requires visual detail that aggressive compression could discard; classification may work with a less detailed representation. Reducing the number of image tokens is intended to speed up inference, particularly on CPUs, which have less parallel compute available than GPUs.

The closing update announces a seed round, with Felicis and Ascend shown on the slide and the GitHub Fund added verbally. Funding would support more GPUs and, more importantly for the founder’s workload, additional people. Korrapati invites potential hires to contact him through the website or Twitter and previews another release planned for later that summer. These are the plans announced at the time of the recording.

“A few updates” slide lists a seed round, hiring, and a summer update, with Felicis and Ascend logos and a speaker inset.Open full source frame
Updates announced in the talk: a seed round, hiring, and a big update that summer.
15:5516:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:55 · section reference included

An offline webcam loop, redirected by a question

For the live demonstration, Korrapati turns off Wi-Fi and states that the system is running entirely locally. A webcam supplies images to Moondream in a repeating loop, and the model describes what it sees. The question applied to those incoming images can be changed while the demonstration runs.

He changes the task to “Is the person wearing glasses?” Response formatting needs its own instruction: asking the visual question alone does not necessarily produce a short answer. He refines the prompt with “Answer with a single word.” A “Yes” response follows, and he removes his glasses to test the changed scene.

At a request to point the webcam toward the audience, he turns it outward and asks, “How many people?” The closing interaction uses the same local image loop for scene description, an attribute question and a request to count—changing the requested behavior through language.

17:3017:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:30 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…