AI Engineer World's Fair 2026
Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face
Slide chapters
22 slide and demo groups · Timestamps are approximate. 6 additional groups were omitted from this pass.
Read the talk
Skill issue: stop deploying vision language models, use them with Skills
Merve Noyan’s agent-assisted workflow uses vision language models to create and judge training labels, then trains a smaller detector for deployment. The hard parts are deciding which annotations to keep, which prompts to trust, and which image transformations preserve the task.
From a talk by Merve Noyan
At a glance
Ideas worth remembering
Use VLMs to create and review supervision, then train a task-specific detector for the runtime application.
Overlaying proposed boxes on images lets judges inspect the region alongside its label description. Keeping an example when either judge approves preserved training data in Noyan’s imbalanced-judge experiments.
Human judgment remains necessary for class descriptions and augmentation choices: a mirrored sign or recolored traffic light can invalidate the original label.
The document detector finding a signature missed by its labeling VLM illustrates generalization beyond an individual teacher prediction; it is a qualitative result without equivalent ground-truth evaluation.
Start with the camera’s frame rate
A camera application needs to keep up with incoming frames. Sending every image to a vision language model makes that requirement difficult, especially on modest hardware. Merve Noyan, a computer vision practitioner at Hugging Face who has written a book on VLMs, opens with an intentionally provocative request: stop calling them directly for everything. Her target is a vision application that runs end to end, with a model suited to its runtime job.
The practical goal is roughly 30–40 frames per second on what she jokingly calls a toaster, whether the task is classification or instance segmentation. A task-specific model such as RF-DETR is her preferred route. This is a deployment argument about speed and task fit; the recording’s examples do not establish a universal performance advantage across every VLM, device, and vision problem.
Model selection also includes the license. Noyan uses developers’ recurring requests for YOLO as an example of popularity outrunning attention to deployment terms, and favors Apache 2.0 models. Her warning is to inspect the particular model’s license before choosing it; the talk’s brief license discussion does not establish payment obligations for every model sold or distributed under the YOLO name.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the coding agent computer vision judgment
VibeVision, the toolkit shown on Noyan’s slide and published in her vision-intern repository, starts from a useful division of labor: an agent can call a specialized vision model as a tool. Noyan describes this as distilling her own model-selection habits into something a coding agent can use. The agent may write good software while remaining, in her words, a “clueless computer vision engineer.” Supplying tools and task knowledge reduces the number of vision decisions it has to invent.
Her selection criteria are deliberately practical:
- License: Check the terms before adopting the model, with Apache 2.0 and MIT among her preferred choices.
- Performance for the size: Compare benchmark results in light of model size and architecture, rather than choosing the largest model by default.
- Vibes: Keep room for the practitioner’s judgment about which models are worth using.
The more involved half is “vibe training.” With labeled images, a developer can go straight to training. With images alone, there are two additional jobs: create annotations and evaluate them. A VLM becomes the labeler, other VLMs become judges, and the retained annotations become training data for the detector. The expensive general-purpose visual reasoning moves into dataset preparation.
That makes the coding agent’s job a long-running workflow: oversee labeling, judging, and training rather than produce a single script and disappear. It can run locally or use Hugging Face infrastructure. Jobs handle one-off batch processing and training; Inference Providers route serverless model calls; buckets hold intermediate data alongside dataset and model repositories. Smaller, more capable vision models make the annotation stages cheaper, while the agent keeps the stages moving.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Judge the box on the image
The pipeline labels an image dataset with a nine-billion-parameter VLM, then sends the annotations to two smaller judges. Their decisions are merged before training RF-DETR medium or large. Noyan avoids asking the judges for numerical quality scores: in the research she discusses, scores from differently sized models do not provide a useful common scale. The decision is instead whether to approve an annotation.
The important representation change happens between labeling and judging. The labeler emits bounding-box coordinates as tokens. Those coordinates are used to draw boxes over the original image, and the judges receive that image with labels and label descriptions. This lets the judge inspect the proposed region against the visible object and the intended class description. It does not have to infer the spatial relationship from a coordinate list alone. 7:17
What actually reaches the detector’s training set? The flow below separates the visual review input from the annotations used for training. Each judge sees the same proposed boxes and descriptions; approval from either judge is enough to retain an example. The coding agent generates the descriptions, but a human approves them before they guide the judges.
An already labeled dataset can skip annotation and proceed to training. The licensing preference also has a stated exception: Noyan describes one judge as having revenue-dependent commercial terms. Even within a curated pipeline, permission to use a model remains a per-model decision.
From unlabeled images to detector training data
Read the diagram as text
- Image dataset. Images without task labels
- VLM labeler. Proposes labels and bounding boxes
- Render the proposed boxes. Overlay boxes on the image and attach label descriptions
- Human-approved descriptions. Coding agent drafts what each label means
- Smaller judge A. judges — Approve or reject
- Smaller judge B. judges — Approve or reject
- Retain if either approves. Minimum agreement preserves more training examples
- Train RF-DETR. Medium or large detector learns from retained annotations
- Image dataset → VLM labeler: Label.
- VLM labeler → Render the proposed boxes: Draw emitted coordinates.
- VLM labeler → Retain if either approves: Proposed labels and box coordinates to filter.
- Human-approved descriptions → Render the proposed boxes: Add class meaning.
- Render the proposed boxes → Smaller judge A: Visual review.
- Render the proposed boxes → Smaller judge B: Visual review.
- Smaller judge A → Retain if either approves: Verdict.
- Smaller judge B → Retain if either approves: Verdict.
- Retain if either approves → Train RF-DETR: Filtered annotations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A few dollars, then two different tests
Noyan reports about three to four dollars for an entire experimental pipeline run. She uses serverless inference for labeling, Hugging Face Jobs for judging, and an L4 for training. That figure belongs to her experiments: without a dataset size or complete workload specification, it is not a general price estimate. Hardware choice also reflects impatience and a preference for a large batch size; she says the small detector can be trained locally or on other hardware.
Road-sign detection provides an external check on the generated labels because the dataset already has ground-truth annotations. Noyan also runs the labeling VLM on the test set and compares against both its pseudo-annotations and the ground truth. She reports a good mean average precision, with a gap between the comparisons. The metric wording does not support a precise score or threshold, but the distinction matters: agreement with the labeler and agreement with ground truth measure different things.
Document parsing asks a different question: can the detector learn to locate images, tables, and signatures from generated annotations? Noyan repurposes a document question-answering dataset for this extraction task, so she lacks equivalent ground-truth evaluation. The result is a qualitative example of generalization rather than a measured document-parsing accuracy.
The signature example makes the change concrete. The training workflow first creates document-region labels, filters them through judges, and trains RF-DETR. On a test document, the labeling VLM misses a signature, while the trained detector finds it. Learning from generated labels has produced a detector that can recognize a region its teacher misses on that image. Noyan credits RF-DETR’s backbone for this behavior; the example illustrates why noisy supervision need not make the student reproduce every individual teacher error. 10:57
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filtering can starve the model; augmentation can change the answer
Two judges do not automatically make a balanced review system. One rejects many more examples than the other, depending on the task. Requiring both judges to approve would leave too little training data in Noyan’s experiments, weakening generalization. Her minimum-agreement rule keeps an example when either judge says yes. It trades stricter filtering for more surviving examples, and the observed imbalance is smaller for document parsing.
The recommendation remains task-dependent. Noyan suggests observing how the judges behave on the particular task. Her captioned aside connecting consensus to recall is ambiguous; it does not establish a filtering policy to follow. Judge behavior discussion
The generated judge prompt still needs a person. A coding agent can draft class descriptions and review instructions, but someone must look at the dataset and decide whether those instructions match the intended task. “There’s no escaping that.” Automating the workflow does not remove the need to define what a correct label means.
Training code introduces another kind of mistake: an image transformation can invalidate its label. Noyan catches two examples:
- Horizontal flips of traffic signs: Mirroring can change directional meaning while leaving the original class label attached.
- Color jitter on traffic lights: Changing the color can change the state the model is supposed to recognize.
The coding agent applies familiar augmentation techniques without understanding those consequences. Noyan patches the workflow so augmentation can be explicitly allowed or disabled. The useful question is whether a transformation preserves the answer for this task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose tools for the visual question and the hardware
The toolkit’s other half exposes selected models as agent tools, spanning depth estimation through zero-shot segmentation. The choices draw on benchmark comparisons across model sizes and Noyan’s reading of computer vision papers. This gives the agent a curated set of capabilities rather than expecting a single VLM to perform every visual operation.
Falcon Perception illustrates why capability matters as much as size. The request “segment this red car” identifies an object by an attribute. A request for the red car next to the orange car that is next to the blue car requires resolving relationships before selecting the region. Noyan presents Falcon Perception as able to handle this open-ended referring segmentation, at 600 million parameters with an Apache 2.0 license.
Other tools cover distinct jobs:
- Human-centered analysis: Pose and related models support human keypoint detection and depth tasks.
- Zero-shot detection: Multiple detector sizes let developers choose a smaller alternative when hardware or speed requires it.
- OCR: Models are selected from OCR benchmark results in different sizes.
- Depth estimation: License terms are checked alongside capability; a model family should not be assumed to share one license across sizes.
- Tracking: Supervision and tracker support from Roboflow help follow instances and bounding boxes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When words cannot describe the part
Industrial objects expose a limit of language-based labeling: some parts are difficult to describe in words. Noyan proposes image-guided detection as a possible next step. A reference image shows the object to find, and the detector searches for that object across other images. The example becomes the query, avoiding the need to invent a verbal description precise enough to distinguish the part. This is a planned direction, not a demonstrated industrial result.
A second proposed change would make judging spatial. Instead of asking a judge to accept or reject the labeler’s box, ask it to produce its own box and compare the two using intersection over union: the area shared by the boxes divided by the area covered by either box. That would expose agreement about location directly. Noyan also says segmentation support is in progress.
For further implementation work, Noyan points to a small vision repository covering fine-tuning, quantization, and multimodal models; Transformers task guides with tutorials; and Hugging Face Skills covering computer vision and infrastructure. These serve different needs: learning a model workflow, following a task tutorial, and giving an agent the instructions needed to run training.
The Q&A keeps the priority concrete. Training VLMs themselves in a self-improvement loop sounds exciting to Noyan, but first she wants developers to train task-specific models and deploy them at the edge. She also explains why the coding agent drafts judge prompts: it already has the workflow’s context. That context helps generate instructions; the earlier human review step still determines whether they describe the task correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Shown on the closing resource slide.
Shown on the closing resource slide.
Shown on the closing resource slide.
Read the complete timestamped transcript
22 slide groups. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 6 additional groups were omitted from this pass.
- 0:13
Hello and welcome to this talk on skill
- 0:16
issue. It's actually no longer a skill
- 0:19
issue. So uh by the end of this talk you
- 0:22
will be able to build a lot with the
- 0:25
vision models if you are already not
- 0:26
doing it. So shortly about me, I'm Marv.
Shortly about me
Merve Noyan wants developers to build vision apps using agents, while avoiding direct use of vision language models for simpler tasks. She offers the talk as a baseline to build on.
Text in image
Shortly about me • I work on computer vision/multimodal (for past 3 years, since IDEFICS/Llava times) and lately on on-device/agents at Hugging Face • I love vision language models so much that I even wrote a book on it (find me to get a copy!) • But I don’t want people to directly use vision language models for simpler tasks • I want every single developer to start building vision apps using agents • Good baseline → build on it, get your hands dirty Book cover: O’REILLY Vision Language Models Building VLMs with Hugging Face Merve Noyan, Miquel Farré, Andrés Marafioti & Orr Zohar Foreword by Lucas Beyer
- 0:30
Uh I've been working on computer vision
- 0:33
since Lava in the fix times. Uh and
- 0:37
lately I work on more agents and
- 0:39
ondevice stuff because I'm a bit
- 0:40
fascinated and I love vision language
- 0:43
models so much that I even wrote a book
- 0:45
on it. But I don't want developers to
- 0:47
directly use vision language models
- 0:49
anymore. And uh I want every single
- 0:52
developer to start uh building vision
- 0:56
langu vision applications end to end.
- 0:59
And this talk is going to give you like
- 1:01
a good baseline to do so.
- 1:04
And common behaviors I observe with the
- 1:07
developers is the fact that they try to
- 1:09
use vision language models for
Common behavior I observe
Noyan argues that VLMs can be costly and too slow for real-time, on-device tasks. She contrasts them with a model trained for a specific use case, alongside an RF-DETR demo detecting a person.
Text in image
Common behavior I observe • they use VLMs for everything because natural language is eez: VLMs are too large and costly for what you want to solve 99% of the time • VLMs don’t give you real-time, low cost, low latency: you can’t process 30–40 fps on-device (toasters) (memory-bound) • they’re not as robust as a model actually trained on a specific use case either RF-DETR Interface text: LIVE STREAM streaming #0 person 0.96 Boxes Masks Confidence thr. 0.40 Start streaming Stop Streaming 22.9 fps 16.0s left
- 1:10
everything but they you will never get
- 1:14
real time. And when I mentioned real
- 1:16
time, it's like you have a toaster and
- 1:19
you get like 30 40 fps uh on it like
- 1:22
whatever whether you are building like
- 1:24
image classification, instance
- 1:26
segmentation or whatever.
Common behavior I observe
Noyan argues that VLMs can be costly and too slow for real-time, on-device tasks. She contrasts them with a model trained for a specific use case, alongside an RF-DETR demo detecting a person.
Text in image
Common behavior I observe • they use VLMs for everything because natural language is eez: VLMs are too large and costly for what you want to solve 99% of the time • VLMs don’t give you real-time, low cost, low latency: you can’t process 30–40 fps on-device (toasters) (memory-bound) • they’re not as robust as a model actually trained on a specific use case either RF-DETR Interface text: LIVE STREAM streaming #0 person 0.96 Boxes Masks Confidence thr. 0.40 Start streaming Stop Streaming 22.9 fps 16.0s left
- 1:29
And um they are not super robust in
- 1:32
terms of like if you were to train like
- 1:34
an RFDTR which Joseph spoke in the I
- 1:37
spoke about in the first talk um it will
- 1:41
always outperform your vision language
- 1:43
model and I'm going to prove it today.
- 1:47
And uh on the right hand side you can
- 1:49
actually see me uh doing stuff with
- 1:51
RFDTR.
- 1:53
Um and another thing is that they don't
Common pitfalls I observe with developers
Noyan points to repeated YOLO questions as a sign that developers may overlook model licenses, and urges them to consider Apache 2.0 alternatives.
Text in image
Common pitfalls I observe with developers devs don’t check licenses: so many people use YOLO etc (lovely models but NC), if you want to use it you have to pay for it, we have more alternatives, easy to use with transformers just because a model is open-source doesn’t mean you can use it! Replying to @skalskip92 @GoogleAI and 2 others Interesting. Is it better than fine tuned usual detection model? (e.g. YOLO) Replying to @mervenoyann and @huggingface Wow, I don't know that RF-DETR exist in HF. I want to learn my custom dataset learn on RF-DETR and compare with YOLO Interface text: 1 2 262
- 1:56
read the licenses. Everybody's like
- 1:58
whenever I post something about object
- 2:00
detection, they always ask me about
- 2:02
YOLO. Like YOLO is a good model. Uh but
- 2:04
it's a it has AGPL 3.0 license I think.
- 2:08
And I could swear on my life that there
- 2:10
is like some developers that actually
- 2:11
deploy it without knowing that they have
- 2:13
to pay for it. Um so yeah like uh I want
- 2:18
you to migrate to Apache 2.0 models
- 2:22
today.
VibeVision
Noyan introduces VibeVision, which she built around letting agents code and ship computer vision projects.
Text in image
VibeVision // agentic computer vision Build vision projects the agentic way: let the agents code and ship. Hugging Face detec[illegible] agent
- 2:24
And for this I have built something
- 2:26
called web vision and it's kind of
- 2:28
inspired by this post by Mazar.
Inspiration
A post inspired VibeVision: Gemma 4 calls SAM 3.1 to segment vehicles in a parking lot, finding 64 vehicles and then 23 white ones. The post says both models run locally on a MacBook.
Text in image
Inspiration Maziyar PANAHI @MaziyarPanahi Gemma 4 looks at a parking lot. Decides what to ask. Calls SAM 3.1. "Segment all vehicles." 64 found. "Now just the white ones." 23 found. One model reasoning and orchestrating. One model executing. Both running locally on a MacBook. MLX. No cloud. No API. Interface text: 0:10
- 2:30
Basically what he does is that he gives
- 2:33
um SAM 3.1 model as a tool to Gemma 4 to
- 2:37
call and I find it super impressive. Um
- 2:40
and today I built like a tool kit uh
- 2:44
where you can do this with like even
- 2:46
more things.
VibeVision: Distilling myself
Noyan selects models as tools for an agent, prioritizing licenses that allow commercial use. She also weighs performance on unsaturated benchmarks against model size—and checks the vibes.
Text in image
VibeVision: Distilling myself • my favorite models as tools for different tasks • I check following ◦ License (biggest prio): doesn’t have to be A2.0 and MIT, has to allow commercial use ◦ Performance on unsaturated benchmarks vs model size ◦ Vibes
- 2:49
So I'm kind of distilling myself. Um
- 2:52
first off uh this toolkit has my
- 2:55
favorite models as tools uh so that you
- 2:57
can give it to your agent because your
- 2:59
coding agent is a bit of a clueless
- 3:02
computer vision engineer basically. Um
- 3:05
and when I distill myself basically like
- 3:07
whenever I pick a model I always check
- 3:09
the following. So first up the license
- 3:11
is the biggest priority for me. It has
- 3:14
to be Apache 2.2 MIT or something
- 3:16
non-commercial. Uh secondly uh the
- 3:20
performance has to be on par uh
- 3:22
depending on its size or architectural
- 3:24
choices. So I check from the benchmarks
- 3:28
uh whenever a model from a computer
- 3:30
vision conference comes out and third
- 3:32
vibes obviously
Building an object detection model
With labelled images, a developer or agent can train directly. With images alone, Noyan proposes using a VLM to label them, judging the labels, then training the model.
Text in image
VibeVision I put myself in developers’ shoes, what is the easiest way to build an object detection model? • If I have labelled images → just train (yourself or agent), easy • If I have images only → label with VLM, judge labels, train model. How does the pipeline look like?
- 3:35
and so this toolkit has a second part to
- 3:39
it and it's a bit like a vibe training
- 3:41
part which is the most exciting part. So
- 3:44
I will go through it first. So I put
- 3:47
myself in developers shoes to build like
- 3:49
a vision application. Um if I have
- 3:52
labeled images easy I can just train a
- 3:55
model or like if I I can give my
- 3:57
computer computer vision agent some
- 4:00
tutorials uh to do that because it's all
- 4:02
out there like we built transformers for
- 4:05
it. Uh but if I have images only I have
- 4:08
to annotate uh and then uh evaluate the
- 4:11
annotations and then just train a model.
- 4:14
But how can you do it on scale? You can
- 4:16
actually use a vision language model as
- 4:18
a labeler and the vision language models
- 4:21
as judge and then train what you want.
- 4:24
But how does this pipeline look like?
VibeVision workflow and infrastructure
VibeVision uses VLMs to label images and judge labels before training a model. The workflow supports local inference and training, remote jobs, inference providers, and intermediate data storage.
Text in image
VibeVision: Distilling myself • I built a generic model training, agent-friendly workflow, works on different domains (for baseline to get you started!) ◦ VLM for labelling → VLMs-as-judges → training • infra support ◦ local model inference/training ◦ Jobs for remote batch processing/training ◦ Inference Providers for labelling/judging ◦ Buckets to dump intermediate data
- 4:28
So basically I built this and it has
- 4:32
like it has VLM for labeling VLM as a
- 4:35
judge and then training. Uh it's a bit
- 4:38
of a long horizon tasks for coding
- 4:40
agents and it has infra many
- 4:42
infrastructure support. You can do this
- 4:44
locally. You can do this uh remotely. So
- 4:47
basically it runs on hugging face
- 4:48
infrastructure. We have jobs that allow
- 4:51
you to do oneoff batch processing or
- 4:54
training. We also have like a serverless
- 4:57
routing system called inference
- 4:59
providers where you can use multiple
- 5:01
providers and you we also have like
- 5:04
buckets to dump intermediate data on top
- 5:06
of the data set repositories, model
- 5:08
repositories and so on.
What enables this work?
Long-horizon agents can oversee labeling and training, while smaller vision-language models reduce labeling costs. RF-DETR and refactored Transformers v5 support the workflow; RF-DETR-Seg is a work in progress.
Text in image
What enables this work? • We have better agents for long horizon tasks • We owe this to RF-DETR, and for segmentation [WIP] RF-DETR-Seg • Smaller yet more capable vision language models for labelling so we don’t pay a ton • transformers DETRs are refactored + v5 is more robust (RF-DETR family is now available as well as many other models)
- 5:11
Um but what enables this work? First
- 5:14
off, my favorite model, RFDTR, RFDTR
- 5:17
segmentation. I'm currently working on
- 5:19
segmentation at the moment. Um, we have
- 5:22
better agents for long horizon tasks.
- 5:25
Uh, that, you know, you have to babysit
- 5:27
the labeling process, the training
- 5:30
process, etc. Uh, and smaller yet more
- 5:34
capable vision models enable you to
- 5:36
label stuff very cheaply. And also with
- 5:40
transformers we went for V5 refactoring
- 5:43
etc. So like it performs better for
- 5:45
vision models at the moment.
VLM-labeled detection training pipeline
Qwen labels images, then Gemma and LFM judge the detections. A vote keeps agreed labels for RF-DETR training, while a coding agent supervises each stage.
Text in image
vision-agent — VLM-labeled detection training pipeline ORCHESTRATOR · SUPERVISES EVERY STAGE Any coding agent of your choice Drives & babysits the run — never labels, never judges. Input data HF dataset · DocVQA Local dir · COCO INFERENCE PROVIDERS · remote 1 Label · detect label_qwen.py LABELER · VLM Qwen/Qwen3.5-9B Qwen · 9.65B Detects objects → boxes + class labels. Labeled dataset + detections HF JOBS · managed compute 2 Judge A · JUDGE judge_one.py google/gemma-4-E4B-it Google · 8.0B VLM-AS-JUDGE ENSEMBLE · SCORES 0-1 2 Judge B · JUDGE judge_one.py LiquidAI/LFM2.5-VL-1.6B Liquid AI · 1.6B 3 Merge · ensemble merge_judges.py MODEL — ensemble vote logic cpu-upgrade Vote across judges → keep agreed detections. Judged dataset filtered + verdicts 4 Train · fine-tune train_rfdetr_job.py DETECTOR Roboflow/rf-detr-base RF-DETR GPU Fine-tunes the object detector on kept labels. Trained RF-DETR checkpoint Labelling with Qwen3.5-9B LFM2.5-VL and Gemma-4 as judges merge judgements train RF-DETR
- 5:47
And this is how the pipeline actually
- 5:49
looks like. So first up I label the data
- 5:53
set like I take an image data set like
- 5:55
any image data set. I label that image
- 5:57
data set with Q1 3.59B
- 6:00
and then I pass the labeled data set to
- 6:03
two judges. First one is Gemma 4 E4B
- 6:07
which is like an 8B judge and then the
- 6:10
second one is LFM 2.5VL which is nearly
- 6:13
2B uh it's relatively smaller basically
- 6:16
I checked the research and it's better
- 6:18
to have like an ensemble smaller judges
- 6:21
and on top of it I merged the judgments
- 6:24
so I also checked the research around it
- 6:27
and most of the people ask the VLM or
- 6:30
LLM to assign some score to it but those
- 6:33
scores absolutely absolutely don't work
- 6:35
especially if your models are of
- 6:36
different size with judges. Um and then
- 6:40
I pass it to train RFDTR medium or
- 6:43
large. So I I chatted a bit with Rob
- 6:46
people and they encouraged to use that
- 6:49
and I'm going it it actually works. I'm
- 6:51
going to show you very shortly. U but
Vibe-training pipeline
A coding agent can run the labeling, judging and training pipeline from a dataset prompt. The example pauses for approval of the generated judge descriptions; already-labeled data can go straight to training.
Text in image
Vibe-training Pipeline • Just clone the repository and prompt “Run the annotation and training in this project on username/objects dataset to detect objects” to your coding agent › can you run e2e pipeline in this repository on this dataset where you need to detect soda bottles? Francesco/soda-bottles use recommended setup • I'll run the e2e pipeline using the dedicated vision-e2e-runner agent, which handles the label → judge → train flow and halts at the one approval gate (the generated judge descriptions).
- 6:54
how does it work? So you take the
- 6:56
repository and then you just ask it okay
- 6:59
can you train it? Can you do the
- 7:01
training on this data set on hub? And
- 7:05
then it will start like if if the data
- 7:08
set has labels like you can actually
- 7:10
just get to training but if it doesn't
- 7:13
have it you can just uh start annotating
- 7:17
and I basically like the trick is I I
- 7:21
pass the overlaid bounding boxes on
- 7:23
images to the judge. So basically Qan
- 7:26
technically outputs bounding boxes as
- 7:28
tokens. I don't pass them. I just
- 7:30
overlay the bounding boxes and I pass
- 7:32
that image on top of some labels and
- 7:35
label descriptions and I say okay if
- 7:38
this label description has bounding box
- 7:40
on it then just um tell me if you
- 7:44
approve or not and then the judge I I
- 7:47
merge the judge verdicts over minimum
- 7:50
agreement and not consensus which I will
- 7:53
come to why I did that that way and
- 7:56
these label descriptions are also
- 7:58
generated by coding agents. and you just
- 8:00
approve it as a human being.
Models used in the pipeline
Noyan uses Qwen3.5-9B for one-off labelling and smaller models as judges. RF-DETR handles detection; the agent workflow was built with Opus 4.8 and then run with GLM-5.2.
Text in image
Models used in pipeline • Multiple smaller (than labeller) judges perform well (based on research and common sense) • Qwen3.5-9B: 9B model, ~19 GB for one-off labelling • Gemma-4-E4B-it: 8B params, 15.1 GB at BF16 • LFM2.5-VL-1.6B: edge focused dense model ~1.6B params, takes 2.34 GB in bf16 • RF-DETR-medium: 33.7M params, sota real-time detection model by Roboflow, comes in various sizes, DINOv2 backbone • For agents, I initially built with Opus 4.8 and then ran the workflow with GLM-5.2 *All Apache 2.0 licensed except LFM2.5-VL
- 8:03
And the the models that I used in this
- 8:06
pipeline all have Apache 2.0 of licenses
- 8:09
except for LFM model which has type of
- 8:13
license where you are kind of um if you
- 8:16
have like certain amount of revenue
- 8:19
after that you pay for it but you can
- 8:21
comfortably use it it's large and for
- 8:25
the coding agents that are babysitting
- 8:26
this pipeline uh I initially built with
Models used in the pipeline
Noyan uses Qwen3.5-9B for one-off labelling and smaller models as judges. RF-DETR handles detection; the agent workflow was built with Opus 4.8 and then run with GLM-5.2.
Text in image
Models used in pipeline • Multiple smaller (than labeller) judges perform well (based on research and common sense) • Qwen3.5-9B: 9B model, ~19 GB for one-off labelling • Gemma-4-E4B-it: 8B params, 15.1 GB at BF16 • LFM2.5-VL-1.6B: edge focused dense model ~1.6B params, takes 2.34 GB in bf16 • RF-DETR-medium: 33.7M params, sota real-time detection model by Roboflow, comes in various sizes, DINOv2 backbone • For agents, I initially built with Opus 4.8 and then ran the workflow with GLM-5.2 *All Apache 2.0 licensed except LFM2.5-VL
- 8:29
oppus 4.6 uh 8 and then ran the workflow
- 8:33
with the G gl GLM 5.2 2 which is doing a
- 8:36
good job on long horizon tasks to be
- 8:38
honest
Infrastructure and costs
Noyan estimates ~$3 to process 1.4k images end to end: DeepInfra labels them for $0.16, parallel HF Jobs run the judges, and L4 training costs about $1. Together is faster but costs more.
Text in image
Infra • note: I work at HF, have a ton of creds and I’m impatient, but it costs ~$3 to run e2e (1.4k images) • Qwen3.5-9B on Inference Providers, serverless ◦ Labelling 1.4k images: $0.16 (with DeepInfra, if you want faster use Together) • Gemma-4 and LFM2.5-VL on parallel HF Jobs (batch inference charged hourly) ◦ I put Gemma-4 to L40S (~$1, ~$1.8/hr) (wallclock, would fit to L4 with bsz = 8) ◦ LFM to L4 ($0.47, ~$0.8/hr) • Training (again, I’m impatient): l4 (~$1) Model | Provider | Input $/1M | Output $/1M | Context | Latency(s) Qwen/Qwen3.5-9B | together (fastest) | $0.17 | $0.25 | 262,144 | 0.35 Qwen/Qwen3.5-9B | ovhcloud | $0.12 | $0.18 | 262,144 | 0.40 Qwen/Qwen3.5-9B | deepinfra (cheapest) | $0.10 | $0.15 | 262,144 | 1.95
- 8:40
and for infra I actually work at hugging
- 8:42
face I have a lot of compute credits and
- 8:45
I'm super impatient in life so like uh I
- 8:49
use a good amount of hardware for
- 8:51
experimentations but I benchmarked it
- 8:54
and overall it takes like three four
- 8:57
dollars if you want to run this entire
- 8:59
pipeline to train models which to me is
- 9:02
crazy initially for Q1 3.5 5 I used
- 9:05
serverless because I was like okay this
- 9:08
is convenient and it's super cheap so I
- 9:10
use deep infra which is super super
- 9:13
cheap if you were to use together it's
- 9:15
better if you do batch processing over
- 9:18
jobs and then for the judging I used
- 9:21
hugging face jobs which cost less and
- 9:24
then for training again I used like an
- 9:27
L4 but the model is super small like
Infrastructure and costs
Noyan estimates ~$3 to process 1.4k images end to end: DeepInfra labels them for $0.16, parallel HF Jobs run the judges, and L4 training costs about $1. Together is faster but costs more.
Text in image
Infra • note: I work at HF, have a ton of creds and I’m impatient, but it costs ~$3 to run e2e (1.4k images) • Qwen3.5-9B on Inference Providers, serverless ◦ Labelling 1.4k images: $0.16 (with DeepInfra, if you want faster use Together) • Gemma-4 and LFM2.5-VL on parallel HF Jobs (batch inference charged hourly) ◦ I put Gemma-4 to L40S (~$1, ~$1.8/hr) (wallclock, would fit to L4 with bsz = 8) ◦ LFM to L4 ($0.47, ~$0.8/hr) • Training (again, I’m impatient): l4 (~$1) Model | Provider | Input $/1M | Output $/1M | Context | Latency(s) Qwen/Qwen3.5-9B | together (fastest) | $0.17 | $0.25 | 262,144 | 0.35 Qwen/Qwen3.5-9B | ovhcloud | $0.12 | $0.18 | 262,144 | 0.40 Qwen/Qwen3.5-9B | deepinfra (cheapest) | $0.10 | $0.15 | 262,144 | 1.95
- 9:29
RFDTR is super small and you can just
- 9:32
use something else you can do it locally
- 9:34
if you wanted to. I'm just impatient. I
- 9:36
want a big batch size. So yeah,
VibeVision test problems
Noyan compared road-sign detection against existing ground-truth labels. For document parsing, she wanted to extract images, tables and signatures from DocVQA documents.
Text in image
VibeVision Tested on various problems • Roadsign detection: ground-truth labels available, compared against ground truth • Document parsing: extracting non-text media from DocVQA dataset
- 9:40
and I tested in two problems. First of
- 9:42
the road sign detection. Uh secondly,
- 9:46
document parsing. For the road sign
- 9:48
detection one, I already have the
- 9:49
labels. So like I actually compared
- 9:52
against the ground truth annotations if
- 9:54
my pipeline works or not. And for
- 9:56
document parsing, I actually uh couldn't
- 9:59
do it because basically I use like a
- 10:01
docqa data set and I the problem is like
- 10:04
I want to extract the images, tables, I
- 10:07
don't know signatures and stuff. So it's
- 10:09
like a novel task and I wanted to see if
- 10:11
RFDTR can actually learn it.
First road sign detection results
The road-sign detector scores 0.9693 mAP@50 against Qwen pseudo-labels and 0.8028 against human annotations. Its ROC-AUC against human ground truth is 0.987.
Text in image
First run results: it works! railway_crossing 0.73 T_intersection | 0.53 do_not_turn | 0.50 do_not_turn | 0.53 tldr; mAP is “how good is the detector”, ROC-AUC measures "do the model's confidence scores rank true detections above false ones" • pseudo-labels by Qwen (judged): map@50=0.9693 • human GT annotations: map@50=0.8028 • ROC-AUC (human GT): 0.987
- 10:15
So first result it works. So yay. um we
- 10:19
have like a good uh mean average
- 10:21
precision over um 50 um and I compare it
- 10:26
against the basically I have a test set
First road sign detection results
The road-sign detector scores 0.9693 mAP@50 against Qwen pseudo-labels and 0.8028 against human annotations. Its ROC-AUC against human ground truth is 0.987.
Text in image
First run results: it works! railway_crossing 0.73 T_intersection | 0.53 do_not_turn | 0.50 do_not_turn | 0.53 tldr; mAP is “how good is the detector”, ROC-AUC measures "do the model's confidence scores rank true detections above false ones" • pseudo-labels by Qwen (judged): map@50=0.9693 • human GT annotations: map@50=0.8028 • ROC-AUC (human GT): 0.987
- 10:29
and I take that test set and pass
- 10:31
through QN and then I compare against
- 10:34
the pseudo notations and the ground root
- 10:37
annotations of that test set there is a
- 10:40
bit of a gap but it's kind of expected
- 10:42
because it's learned from QN and also uh
- 10:45
rock oak is also like a good value to be
- 10:48
frank for like such use case. Um and for
- 10:53
the document parsing it actually
- 10:55
generalizes which to me is crazy.
- 10:57
Basically uh here the trained model
- 11:00
output you can see that it detected the
- 11:03
signature. Meanwhile the QN annotation
- 11:06
of that test set missed it. So I would
- 11:09
like to say it actually generalizes very
- 11:11
well as well. uh but we owe this to like
- 11:14
the how good RFDTR is as a backbone in a
- 11:18
way
Document image detection comparison
The model’s three image boxes closely match the Qwen3.5 annotations on the same apparel page, including the two photographs below the main fashion image.
Text in image
Qwen3.5 annotation Lifestyle Apparel WILLS LIFESTYLE Synonymous with elegance and style that is effortless and chic, Wills Lifestyle presents a premium fashion wardrobe for men and women. With a tempting choice of Wills Classic formal wear, Wills Sport relaxed wear, Wills Clublife evening wear and Wills Signature designer wear, Wills Lifestyle offers a truly delightful shopping experience for the discerning customer. WILLS CLASSIC WILLS SPORT WILLS clublife WILLS SIGNATURE johnplayers jeans johnplayers jeans is the new language for the fashion-forward youth, featuring an innovative collection of denims and casual wear in edgy styling, bold patterns and contemporary fits. JOHN PLAYERS John Players embodies the spirit of the modern youth that is playful, fashionable and cool. John Players presents a complete wardrobe incorporating the most contemporary trends in an exciting mix of colours, styling and fits. Model Output Lifestyle Apparel WILLS LIFESTYLE Synonymous with elegance and style that is effortless and chic, Wills Lifestyle presents a premium fashion wardrobe for men and women. With a tempting choice of Wills Classic formal wear, Wills Sport relaxed wear, Wills Clublife evening wear and Wills Signature designer wear, Wills Lifestyle offers a truly delightful shopping experience for the discerning customer. WILLS CLASSIC WILLS SPORT WILLS clublife WILLS SIGNATURE johnplayers jeans johnplayers jeans is the new language for the fashion-forward youth, featuring an innovative collection of denims and casual wear in edgy styling, bold patterns and contemporary fits. JOHN PLAYERS John Players embodies the spirit of the modern youth that is playful, fashionable and cool. John Players presents a complete wardrobe incorporating the most contemporary trends in an exciting mix of colours, styling and fits. Detection box labels: [illegible]
- 11:20
and here you can also see how it uh
- 11:22
captures the images technically
- 11:25
um and it's like oneonone
- 11:28
and while I was building this I actually
- 11:30
noticed that I was clueless about
- 11:32
building with vision agents um so I have
- 11:35
bunch of findings around it um so first
- 11:38
up there is a huge judge imbalance so
- 11:40
for depending on the problem. LFM tends
- 11:44
to reject a lot. That's why I couldn't
- 11:46
take the consensus because if I were to
- 11:48
if I were to eliminate everything that
- 11:50
both LFM and Gemma agreed to remove, um
- 11:55
I would left with very very little
- 11:57
number of examples which would leave me
- 11:59
with very poor generalization. So what I
- 12:01
did was that okay, if one of them says
- 12:03
yes, I'm going to take that example and
- 12:05
nevertheless it worked well. If you have
- 12:07
a large data set and if you care about
- 12:09
the recall, I suggest that you take the
- 12:13
consensus or like just observe for
- 12:16
document parsing the gap isn't as big.
Judging prompt generation
Generated judge descriptions need human approval: prompts for simple labels can be too long, while descriptions of specific things can fall short. The slide shows a verbose soda-bottle description.
Text in image
Judging Prompt Generation (Metaprompt) • Asking LLM to generate labels for judges can sometimes backfire depending on class labels in the dataset repository • We leave it to human to approve label descriptions, e.g. for simple things prompt gets too long, for specific things it falls short Approval gate: generated judge description { "soda bottle": "a glass or plastic beverage container featuring a cylindrical body that tapers into a narrower neck, typically topped with a metal or plastic cap and often wrapped with a paper or plastic label displaying brand text and logos" }- 12:21
And secondly, uh the prompt generation
- 12:24
is a bit hard. So like this is the only
- 12:27
part where as a human you have to
Judging prompt generation
Generated judge descriptions need human approval: prompts for simple labels can be too long, while descriptions of specific things can fall short. The slide shows a verbose soda-bottle description.
Text in image
Judging Prompt Generation (Metaprompt) • Asking LLM to generate labels for judges can sometimes backfire depending on class labels in the dataset repository • We leave it to human to approve label descriptions, e.g. for simple things prompt gets too long, for specific things it falls short Approval gate: generated judge description { "soda bottle": "a glass or plastic beverage container featuring a cylindrical body that tapers into a narrower neck, typically topped with a metal or plastic cap and often wrapped with a paper or plastic label displaying brand text and logos" }- 12:29
approve uh okay the model generates the
- 12:32
prompts for you for the judge uh and
- 12:35
then you will say okay this I I approve
- 12:37
this because you need to to take it
- 12:40
still take a look at it uh take a look
- 12:42
at your data set a little there's no
- 12:44
escaping that
Augmentation
A coding agent flipped road signs and applied color jitter to traffic lights, risking corrupted training data. Noyan patched the augmentation behavior later.
Text in image
Augmentation • Sometimes coding agents augment the dataset where it shouldn’t, I have patched this later • Example: roadsigns 🚦 🚧 ◦ the training pipeline flips the images horizontally/vertically which damages “turn right/left” ◦ applies color jitter to traffic lights 😁
- 12:46
um and thirdly and this is super
- 12:50
interesting because your co your coding
- 12:53
agent despite no matter how uh good it
- 12:56
is. Like you take OPUS 4.8 which is like
- 12:59
a very good coding agent. Um it's
- 13:02
clueless as a computer vision engineer
- 13:05
as well as it misses common sense. For
- 13:07
instance like it was doing like
- 13:08
horizontal flip over the traffic signs
- 13:11
or it was doing like jitter over the
- 13:14
traffic lights which will definitely
- 13:17
corrupt your data set and break it. So I
- 13:20
patched this later. So you can just say
- 13:22
okay I don't want to augment or I can
- 13:25
augment. So Angular coding agent will
- 13:27
help you with that.
- 13:30
And lastly the second part of this
- 13:32
toolkit is my preferred models as tools.
- 13:35
Um so this repository covers my favorite
- 13:38
models from depth estimation to zero
- 13:40
shot segmentation.
- 13:42
Uh and this is partially powered by
- 13:44
first of hugging face benchmarks uh
- 13:46
which we recently rolled out maybe a few
- 13:48
months ago. Basically every single
- 13:51
basically we have like a benchmark
- 13:54
leaderboard and over there you have the
- 13:56
open models as well as their evaluation
- 13:59
results and you can compare different
- 14:02
models of different sizes. So um I keep
- 14:06
it up to date as well but like also it's
- 14:09
partially powered by me who likes to
- 14:11
read the um computer vision conference
- 14:14
papers.
Falcon-Perception for segmentation
Noyan recommends 600M-parameter Falcon-Perception for segmentation, saying it can follow positional descriptions such as a red box next to an orange box that SAM cannot.
Text in image
My preferred models as tools • SAM cannot do open-ended referral expression segmentation, e.g. it can segment “red box” but cannot do “red box on the left next to orange box” • Enter Falcon-Perception, 600M model with A2.0 license Comparison columns: Box; Purple box; ACME box Comparison rows: Falcon Perception; SAM3
- 14:16
Um, so I would like to give a shout out
- 14:19
to this model because not a lot of
- 14:20
people know about this. So basically Sam
- 14:23
cannot do open-ended reference se uh
- 14:25
segmentation like you can do like okay
Falcon-Perception for segmentation
Noyan recommends 600M-parameter Falcon-Perception for segmentation, saying it can follow positional descriptions such as a red box next to an orange box that SAM cannot.
Text in image
My preferred models as tools • SAM cannot do open-ended referral expression segmentation, e.g. it can segment “red box” but cannot do “red box on the left next to orange box” • Enter Falcon-Perception, 600M model with A2.0 license Comparison columns: Box; Purple box; ACME box Comparison rows: Falcon Perception; SAM3
- 14:29
segment this red car and it will do it.
- 14:32
But if you say okay the red car next to
- 14:34
the orange car that is next to the blue
- 14:36
car it will not do that. and Falcon
- 14:38
perception which is a model by TIA um
- 14:42
can actually do it and it's only like
- 14:44
600 million parameters with Apache 2.0
- 14:46
license. So this one does the zero shot
- 14:49
segmentation for me and this is like a
- 14:53
non-exhaustive list. So for posing we
- 14:56
have the sapiance family um for the
- 14:59
human centric tasks where you need to do
- 15:02
human koid detection human um depth
- 15:04
estimation and so on and for zero shot
- 15:08
detection uh I have moon dream 3 and mm
- 15:12
grounding dynino which is like a model
- 15:14
with apache 2.0 license is also very
- 15:16
good. It's very small compared to Moon
- 15:18
Dream. I give you the multiple models in
- 15:22
multiple sizes depending on your
- 15:23
hardware that you can pick like if you
- 15:25
want to go fast just pick the tiny
- 15:28
alternative.
- 15:29
Um for OCR I actually took them from the
- 15:32
OCR benchmark in different sizes and for
- 15:36
depth estimation I actually uh
- 15:38
discovered that the large model doesn't
- 15:40
have a non-commercial license and the
- 15:42
rest of them have it. So you can
- 15:43
actually use it. That one has like an
- 15:45
Apache 2.0 license. And it also comes
- 15:48
with um supervision and tracker support.
- 15:50
They are both uh libraries from Roboflow
- 15:52
that um allow you to do tracking of
- 15:56
instances, bounding boxes and so on
Future plans and disclaimers
The pipeline provides a baseline, but industry objects can be hard to describe in natural language. Image-guided detection could help: the illustration uses one visual example to find three matches.
Text in image
Future plans and disclaimers • This pipeline creates a good baseline, no hyperparameter sweep, industry use cases are hard to tackle with natural language • This pipeline can fall short for industry use cases with judges where object is hard to describe, less known • Image guided detection could help with this image-guided detection 1 example → 3 matches visual prompt example guides target image match 0.97 match 0.93 match 0.86 one-shot · detect by visual example, no labels needed
- 16:00
and future plans. So first off, I I
- 16:03
could hear you say, okay, this will
- 16:05
definitely not work for the industry use
- 16:07
cases because industry use cases have
- 16:10
different parts. it has like
- 16:12
nondescribable parts like natural
- 16:14
language is not a good gateway to it. Um
- 16:18
so I think in that sense image guided
- 16:21
detection could help like if you don't
- 16:23
know about image guided detection you
- 16:24
basically have like instance of an image
- 16:27
like a haggi here as an example and then
- 16:30
you ask the model okay detect this
- 16:33
object in this image um across all of
- 16:37
the images. I feel like it could
- 16:39
actually somewhat help in the industry
- 16:40
use cases where you cannot describe it
- 16:43
by like um natural language.
- 16:46
Um as well as I want to try um
- 16:51
intersection over union merger sort of
- 16:54
basically you have like labeled boxes
- 16:57
and then judges boxes like you ask the
- 17:00
judge to actually generate a box and
- 17:02
then you take the intersection over
- 17:03
union instead of asking judge to reject
- 17:06
or accept and I'm working currently
- 17:09
working on the segmentation support
Thanks for listening
Noyan points to smol-vision for model fine-tuning and quantization, Transformers task guides, Hugging Face skills, and the vision-intern repository.
Text in image
Thanks for listening! More resources (for humans and agents): • github: merveenoyan/smol-vision • hf.co/docs/transformers/en/tasks/object_detection (and other task recipes) • github: huggingface/skills • @mervenoyann on X • repo is at merveenoyan/vision-intern smol vision
- 17:12
and thanks for listening if you want to
- 17:14
actually learn more so basically I We
- 17:16
have a small vision repository. It has
- 17:19
everything about like fine-tuning
- 17:21
models, quantizing models, multimodel
- 17:24
models, everything around vision. Uh as
- 17:27
well as transformers task guides. We
- 17:29
keep them up to date has many um
- 17:32
tutorials. Um, we also have hugging face
- 17:35
skills that actually have computer
- 17:38
vision specific skills as well as the
- 17:41
infras skills that you can just do like
- 17:43
a one prompt training. Again, um, and
- 17:47
this is my Twitter profile and this repo
- 17:50
is actually at GitHub Marvin uh, vision
- 17:53
intern. I think I have qu um, time for
- 17:57
one question.
- 17:59
Thank you so much.
- 18:05
Yes,
- 18:05
>> you have plans for training.
- 18:08
>> Um, he's asking if I have uh plans to
- 18:12
train train VLMs themselves like a
- 18:15
self-improvement type of thing. That
- 18:17
would be super exciting. But first, I
- 18:19
want to solve this thing of like
- 18:20
developers actually training task
- 18:23
specific models and then deploying on
- 18:25
edge and then that could come perhaps.
- 18:28
Maybe one more. Yes.
- 18:38
>> Um, not really. I don't think so. I just
- 18:41
use the because I wanted because a
- 18:43
coding agent actually has the context. I
- 18:45
wanted it to generate the prompt.
- 18:49
Maybe one more.
- 18:52
Okay. Thank you so much.
AI Engineer World's Fair
AI Engineer World's Fair title card in white on black.
Text in image
AI Engineer World’s Fair HTTPS://AI.ENGINEER