AI Engineer World's Fair 2026
Infra behind Krea 2 - How to train and serve at scale
Loading slides…
Read the talk
Infra behind Krea 2 - How to train and serve at scale
Krea’s shared GPU cluster gives training priority, absorbs failures with telemetry and checkpoints, and moves inference to external providers when research needs the machines.
From a talk by Gabriel Jorge Menezes
Before you start: Basic familiarity with Kubernetes pods, nodes and scheduling will help with the serving architecture; no diffusion-model mathematics is required.
More room for image exploration
What would it take to make generated images less predictable—to give creatives room for unusual compositions, pixel art, photorealism and playful experiments? That was the starting point for Krea 2, or K2, which Gabriel Jorge Menezes introduces less than a month after its release. Krea trained the generative model in-house from scratch, without a base checkpoint. That scope matters: the surrounding pipeline still uses external pretrained text encoders and autoencoders. The objective was a model built for exploration beyond familiar image styles.
The opening collage makes that range concrete: a suited figure among birds, people running on grass, a pixel-art portrait, a kiwi game scene and cartoon animals. Gabriel describes the release as open source and points viewers to Hugging Face, GitHub and Krea’s production site. The official inference repository describes a community license, so availability should not be read as unrestricted usage rights.
Krea serves two checkpoints in production:
| Checkpoint | Purpose |
|---|---|
| RAW | Pretrained weights for downstream post-training |
| Turbo | Post-trained weights for fast image generation |
Gabriel reports image generation in less than a second with Turbo, without specifying hardware or resolution. The distinction between the checkpoints gives developers a choice between a starting point for further training and an already optimized interactive model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small experiments, increasingly frequent failures
Training ran on thousands of GPUs in one main InfiniBand-connected cluster, but the research began with small ablations. Researchers tested a hypothesis on a few GPUs, let it train briefly, and scaled the ideas that worked. They brought techniques from LLM research into diffusion transformers, while deliberately keeping the model architecture simple.
Small experiments could run for days. As the team expanded to 128, 256 and 512 GPUs, the larger failure surface became visible. Some failures were especially frustrating: an NCCL timeout could stop training even while the available metrics looked healthy. The initial response was to swap nodes and search for a defective machine.
Eventually, the team learned that another restart could be more useful than another hardware swap. Gabriel describes repeated roughly one-hour runs that crashed, followed by runs lasting 12, 16 or 24 hours on the same machines, code and data. That did not make failures harmless; it meant a crash alone was insufficient evidence that a particular node needed replacing.
An unnamed Meta paper provided a comparison for how failures increase at scale. Krea saw a similar pattern, but shorter runs. Gabriel describes large pretraining runs lasting less than eight hours. Every interruption meant GPUs stopped making progress and the release risked slipping. Infrastructure work therefore needed both visibility into real faults and cheap recovery from interruptions. Metrics became the means of supporting researchers without making them diagnose the cluster themselves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure useful work, not just busy GPUs
Temperature was the first useful signal. One GPU running warmer than its peers could throttle, slow the job and introduce unstable behavior. Krea’s policy was to remove GPUs above approximately 78 degrees and ask the provider to replace them; Gabriel does not specify the temperature unit or GPU model. The practical choice was to stop spending research time nursing suspect hardware.
GPU utilization is an activity measurement, not a measure of computational efficiency. NVIDIA defines it as the sampled time during which one or more kernels execute. A GPU can therefore be busy throughout the sample without using its available compute effectively. Krea’s pretraining dashboard showed 100% GPU utilization, which Gabriel says did not mean the GPUs were fully utilized computationally.
The team instead used tensor core utilization as a proxy for useful compute activity. As training progressed through pretraining, mid-training and post-training, image resolution increased. Gabriel recalls resolutions of 128, 256, 512 and 1024 pixels, with tensor core utilization rising as the images required more work. That relationship made the metric useful for interpreting changes in the workload, rather than merely confirming that a kernel was running.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look between the GPUs
Ordinary device metrics left an important blind spot: most of Krea’s observed failures involved communication across nodes. The team’s NVIDIA DCGM export setup lacked the InfiniBand telemetry it needed, so it built custom collection. Throughput was only the beginning. The more developed dashboard included fabric message wait times, packet counts, error counts and different error types. Those measurements helped explain failures that healthy-looking GPU charts could not.
Krea also collected additional NVLink error counters for communication within a node. The missing counters were a limitation of its observed setup, rather than a general absence from NVIDIA’s tooling: archived DCGM documentation already listed NVLink error counters. InfiniBand telemetry was useful more often, but NVLink errors could expose a faulty machine whose GPUs otherwise appeared healthy. That gave the team a concrete reason to replace the machine.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make crashes cheap with frequent checkpoints
Telemetry could identify bad hardware, but training still crashed. The next defense was checkpointing often enough to limit lost progress. Krea initially used Ceph; Gabriel says that deployment broke and left the team unable to trust its stored data. That experience led him to recommend paid storage when the budget permits. It is an account of Krea’s deployment, not a comparison of every Ceph installation with commercial alternatives.
Gabriel reports 1.2 TB/s reads and almost 1 TB/s writes from Krea’s Weka cluster. He reports checkpointing every 20–30 minutes, producing approximately 1 TB of data in under 30 seconds. Hardware and benchmark methodology are unspecified. The operational benefit was that checkpoint writes did not choke training: saving progress became cheap enough to do frequently, reducing how much work each subsequent crash could erase.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let researchers submit jobs, not manage GPUs
The same infrastructure principle shaped job submission: researchers should launch work without first finding available GPUs. Jobs enter a queue and wait until the system can admit them. Krea uses the open-source Kueue project for gang scheduling, so a distributed training job can obtain its required resources together instead of leaving only part of the job running.
There are two separate priority decisions:
| Priority | Decision |
|---|---|
| Kueue workload priority | Which queued training job goes first |
| Kubernetes pod priority | Which pods win resources after admission |
Krea gives training pods high Kubernetes priority. Once admitted, they can preempt lower-priority inference pods. Protecting production therefore requires somewhere else for inference to run; priority alone would simply displace it.
One source of friction was manually configured queue capacity: CPU, NVIDIA GPUs, memory and other resources. Nodes entering maintenance or disappearing from the cluster could leave those numbers out of sync with reality, sometimes breaking gang scheduling. Krea had not automated that synchronization. Gabriel also mentions Kubernetes 1.35 gang scheduling as an alternative the team had not tested. In that version it was alpha and disabled by default, requiring explicit enablement and scheduler configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give training the cluster and move inference elsewhere
Krea runs production and training in one large cluster. Its policy gives training the higher claim on those GPUs because the company sees greater value in using them for research—but the production site still has to work. The solution is to move inference automatically across clusters, external providers and rented GPUs.
The traffic chart in the recording shows local inference in dark green. When someone launches training, inference shifts outward; when training finishes, it shifts back, avoiding unnecessary external spending. Gabriel describes this as seamless for users and researchers: research can consume the local cluster without making someone manually relocate production.
The bridge is Virtual Kubelet, an open-source project that exposes a synthetic node to Kubernetes. The scheduler can place pods on this node using its normal scheduling machinery. What happens behind that node is controlled by Krea’s integration. When local GPUs are occupied, a pod placed there becomes a request for external execution.
Krea’s integration follows a concrete sequence:
- Receive the pod specification assigned to the virtual node.
- Select an external provider through a custom provider interface and selection algorithm, which can account for arrangements such as negotiated prices.
- Translate the pod specification into that provider’s deployment format and deploy it.
- Reconcile state between the external deployment and Kubernetes.
The provider interfaces keep provider-specific details from spreading through the rest of the system. Kubernetes still owns the familiar workload model, including replica counts adjusted by the Horizontal Pod Autoscaler, or HPA.
Failure handling follows the same division of responsibility. If the integration or external execution fails, it reports the pod as failed instead of implementing a separate recovery system. For controller-managed workloads, Kubernetes can create replacement pods. This is distinct from HPA scaling: the HPA adjusts desired replica counts, while an appropriate workload controller handles pod replacement. A failed standalone pod does not acquire a replacement merely because its status changed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prefer local capacity, then return gradually
External execution solves capacity shortages, but it should not consume rental money while local GPUs sit idle. Krea controls access to the virtual node with taints and pod tolerations. When local GPUs are available, the system taints the virtual node so inference pods without a matching toleration cannot newly schedule there. When training consumes local capacity, the system removes that taint, making external placement eligible again. Prometheus metrics drive the availability decision.
For a concrete operational illustration, use vk-external as the virtual node and local-capacity=available:NoSchedule as the gate. Inference pods must lack a matching toleration for this gate to block them. These are the two corresponding state changes:
bash
# Local GPUs are available: block new external placements.
kubectl taint node vk-external \
local-capacity=available:NoSchedule --overwrite
# Training has consumed local GPUs: allow external placements.
kubectl taint node vk-external local-capacity:NoSchedule-
Krea automates this decision rather than requiring an operator to run commands for each training job. Training preempts local inference, the availability signal changes, and removing the taint allows new inference pods onto the virtual node. When training finishes, however, already-running external pods remain. Blocking new placement does not bring those existing workloads home.
The return path uses the Kubernetes Descheduler. Once local GPUs become available, the system restores the virtual-node taint. The descheduler identifies pods that no longer tolerate the node’s taints and evicts them in a controlled progression; workload controllers create replacements, and Kubernetes schedules those replacements onto eligible capacity. This is eviction and replacement, not live migration of a running pod.
Why not apply a NoExecute taint and let Kubernetes evict the external pods directly? Gabriel rejects that approach because it could kick everything out together and take production down. Returning capacity must be gradual even when discovering capacity is immediate. Controlled eviction lets inference return to local GPUs without a simultaneous loss of serving replicas, while reducing external spending. The complete slide combines the placement gate, Prometheus trigger and descheduling return path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A different operating model for research and inference
The system needed debugging and calibration before it could run with little intervention. Once tuned, Gabriel says it changed how Krea conducted research: researchers submitted jobs, jobs waited when capacity was unavailable, and admitted training could use the whole local cluster while production ran elsewhere. GPU availability became something the infrastructure handled rather than something every researcher had to negotiate.
The closing hardware observation is that training and inference did not impose the same demands. Gabriel contrasts diffusion transformers with much larger LLMs, then describes inference continuing on GPUs that were unsuitable for training. His examples escalate from hot GPUs to bus failures and even exploding hardware; the exaggeration conveys how forgiving inference seemed in Krea’s experience, not a policy for operating damaged equipment. The distinction helps explain why a tightly managed training cluster and a broader pool of inference capacity could coexist.
Gabriel closes by inviting people interested in large-scale pretraining and infrastructure for researchers to contact him, and points to Krea’s job listings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official inference code and setup instructions for the RAW and eight-step Turbo checkpoints.
Configure queue priorities separately from Kubernetes pod priorities.
Version-specific documentation for the alpha gang scheduler and its required configuration.
Connect external execution providers to Kubernetes through a virtual node.
Policy-driven pod eviction, including removal of pods that violate node taints.
Further reading
- Krea 2 Technical ReportArticle
Krea's detailed account of model design, training, shared GPU infrastructure and external inference scheduling.
Definitions of GPU activity, temperature and other device monitoring measurements.
Meta's model report includes an analysis of planned and unexpected interruptions during large-scale pretraining.
Full transcript
Loading transcript…