Contents
  1. Visual observations
    1. What vision systems produce
    2. What an image records
      1. Resolution and context
  2. History
    1. A concise history of computer vision
      1. Must-know turning points
  3. Visual representations
    1. Local features and correspondence
    2. Convolutional feature hierarchies
      1. Learning through depth
    3. Patches and spatial interaction
  4. Categories, objects and regions
    1. Categories and visual similarity
    2. Finding objects at different scales
    3. Assignment and duplicate predictions
    4. Dense labels and instance boundaries
    5. Promptable segmentation
  5. Space and visual language
    1. From pixels to scene geometry
    2. Descriptions, referents and relations
    3. Visual features in language responses
  6. Learning and adaptation
    1. What visual supervision rewards
    2. Reconstruction and self-distillation
    3. Adapting reusable features
    4. Preserving annotation meaning
      1. Coordinates and target validity
      2. Named landmarks
  7. Time, motion and identity
    1. Frames, sampling and temporal order
    2. Learning across space and time
    3. Apparent motion
    4. Maintaining object identity
  8. Actions, events and video language
    1. Actions and visible state changes
    2. Answers grounded in video
  9. Ambiguity and changing conditions
    1. Missing information and uncertain interpretations
    2. Capture changes and visual shortcuts
  10. Measuring visual predictions
    1. Scoring categories and detected objects
    2. Scoring masks and spatial claims
      1. Other spatial outputs
    3. Scoring identities and events
      1. Onsets, exposure and processing budgets
  11. Usefulness in the intended setting
    1. Testing the complete visual task
    2. Choosing a practical vision system
      1. Inspection
      2. Mapping and area
      3. Footage and interfaces
  12. Check understanding
  13. Open questions
  14. Selected talks
  15. References
  16. Talk library
← All topics

Vision & video

Computer vision gives software a way to work with visual observations. The useful result might be an object category, the outline of a defect, a distance estimate or the interval when something changed. These outputs require different information. Understanding what the camera records, what the model preserves and what the evaluation measures is the basis for choosing a system that does the intended job.

Visual observations

What vision systems produce

A vision system turns images into task-specific predictions. It computes a representation—an internal encoding of the image that subsequent processing uses to produce the requested output. Naming a component, finding damage and outlining its extent require different distinctions. A representation useful for recognizing the component may not expose the detail needed to locate its defect.

The required output determines the visual task.
TaskOutputWhat it distinguishes
ClassificationImage-level category scoresWhich categories the image supports
DetectionCategories and bounding boxes: rectangular image regionsWhere individual objects lie
Semantic segmentationA category for each pixel, a sample on the image gridWhich regions belong to each category
Instance segmentationA separate mask, recording region membership, for each objectWhich pixels belong to each individual object
Both examples depict a bracket. Inspection asks a different question: whether material is missing, and where. These are synthetic example parts, not before-and-after photographs or model predictions.

Selecting a region from a description adds another requirement. Grounding connects a phrase such as “the cup left of the bowl” to the particular region it concerns. Video adds relationships across observations: tracking associates objects through time, while event understanding identifies what happened and when.

The application then uses these predictions in its workflow. BMW's 2019 production account describes comparing a photographed vehicle badge with order data and sending mismatches to final inspection. Visual recognition supplies information for that decision; the order record and inspection process determine what the result means for the vehicle.

Optical character recognition, or OCR, predicts text from images. Document understanding additionally recovers structures such as reading order, tables and fields. Those tasks are developed in Document Understanding and OCR. Here the focus is the visual information that recognition, localization, spatial interpretation and video understanding require.

What an image records

An image is a measurement of light from a viewpoint. Its values depend on illumination, surface reflectance and orientation as well as scene geometry. A dark patch can represent a dark material or weak illumination. Perspective also makes a nearby small object and a distant larger object occupy similar image regions. Occlusion occurs when one surface hides another; the hidden surface contributes no directly visible detail.

The sensor integrates light over an area and an exposure interval. Longer exposure gathers more light but mixes moving positions into blur. Raising digital gain amplifies the existing measurement and its noise; it does not collect additional photons. Color sensors commonly measure filtered colors at different photosites and estimate colocated red, green and blue values through demosaicing. Thus an RGB image is already a processed measurement.

An image array commonly has height, width and channel dimensions. In this chapter, image coordinates use x rightward and y downward; array access commonly writes the row before the column, as image[y, x]. Encoded file bytes, decoded pixels and model inputs are distinct artifacts. Decoder conventions matter: OpenCV's image reader normally returns BGR channel order and applies embedded orientation unless specified flags override it. A model expecting another convention can receive a different image despite a successful file read.

Synthetic illustration of three information conditions: visible detail, motion-blurred detail, and an occluded edge. These are illustrative views, not measured camera exposures or a controlled performance experiment.

Resolution and context

Resizing changes the available spatial detail. Low-pass filtering before reduction suppresses detail that the smaller grid cannot represent, reducing misleading alias patterns. Enlarging the result interpolates samples; it cannot uniquely restore discarded distinctions.

Consider a 24-pixel-wide object in an image 3,840 pixels wide. Resizing the whole image to width 640 reduces the object to four pixels. A 640-pixel crop processed without reduction retains its 24-pixel width. SAHI, introduced in 2022, applies detection to overlapping crops, maps results back and merges duplicates. Crops preserve detail at the cost of additional passes and less surrounding context; they can split large objects. These dimensions illustrate the mechanism, not a universal detection cutoff.

Constructed comparison: a crop allocates more of the displayed image to the bolt but excludes the surrounding tools. This illustrates the detail–context tradeoff, not a calibrated crop or measured detector improvement.

History

A concise history of computer vision

Computer vision developed along concurrent lines: recovering scene geometry, learning recognition features, locating regions and interpreting motion. The following turning points explain why these approaches coexist. Their mechanisms are developed where each becomes useful later in the chapter.

Must-know turning points

Geometry, learned features and selectable regions

  1. 1963Roberts: 3D from photographsA program converts photographs into line drawings and three-dimensional descriptions of planar-surfaced objects, constrained by perspective, known models and physical support.Sources & context

    Contributors: Lawrence Gilman Roberts

    What changed: Shows how explicit scene assumptions constrain interpretation and permit rendering from another viewpoint.

  2. 1978; developed in the 1982 bookMarr’s 2½-D sketchThe 2½-D sketch represents visible surface geometry relative to the viewer, combining information from processes such as stereo, shading and motion.Sources & context

    Contributors: David Marr

    What changed: Separates describing observed surfaces from recognizing a complete object across viewpoints.

  3. 1981Horn–Schunck optical flowBrightness change alone leaves image displacement underdetermined; a smoothness preference supplies additional constraints for estimating optical flow.Sources & context

    Contributors: Horn and Schunck

    What changed: Provides a motion-estimation mechanism while exposing failures around occlusion and the difference between brightness motion and physical motion.

  4. 1989–1998LeNet → LeNet-5The 1989 postal-digit recognizer learns from normalized images using local connections and shared weights. LeNet-5 develops a trainable convolutional and subsampling hierarchy in 1998.Sources & context

    Contributors: LeCun and colleagues at AT&T Bell Laboratories; LeCun, Bottou, Bengio and Haffner

    What changed: Makes feature extraction part of fitting the recognition system rather than a separate prescribed stage.

  5. 2004SIFTSIFT describes scale- and orientation-normalized neighborhoods with gradient histograms, producing 128-component descriptors.Sources & context

    Contributors: David Lowe

    What changed: Supplies a prescribed representation for matching local appearance across scale and orientation changes, alongside learned recognition.

  6. 2005–2009PASCAL VOC & ImageNetVOC begins in 2005 with shared recognition and localization tasks. ImageNet's 2009 paper describes large, diverse, human-verified category collections organized through WordNet.Sources & context

    Contributors: PASCAL VOC team, described by Everingham and colleagues; Jia Deng and colleagues

    What changed: Makes datasets, annotation meanings and evaluation procedures central components of recognition research.

  7. 2012AlexNetAlexNet combines convolutional features, large labeled collections, GPU computation and regularization for natural-image recognition.Sources & context

    Contributors: Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton

    What changed: Its seven-network competition submission achieves 15.3% top-five test error, compared with 26.2% for the runner-up, demonstrating the combined system's recognition performance.

  8. 2014Two-stream networksA two-stream model combines an RGB network with a network processing stacked horizontal and vertical optical-flow fields.Sources & context

    Contributors: Karen Simonyan and Andrew Zisserman

    What changed: Lets explicit motion complement scene and object appearance for action classification.

  9. 2015; ResNet preprintFCN & ResNetFCNs turn classification networks into spatial category maps with upsampling and fine-feature connections. Separately, ResNet adds shortcut inputs to learned transformations.Sources & context

    Contributors: Jonathan Long, Evan Shelhamer and Trevor Darrell; Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun

    What changed: FCNs support shared whole-image segmentation computation; residual shortcuts improve optimization in comparisons with deeper plain networks.

  10. 2020 preprintVision Transformer (ViT)ViT projects image patches into vectors, adds position information and uses attention to exchange information between tokens.Sources & context

    Contributors: Alexey Dosovitskiy and colleagues

    What changed: Provides a recognition architecture with global patch interaction and a separate classification state.

  11. 2021CLIPCLIP trains coordinated image and text encoders by distinguishing matching image-caption pairs from mismatched pairs.Sources & context

    Contributors: Alec Radford and colleagues

    What changed: Allows candidate descriptions to support classification without fitting a new target-specific classification head.

  12. 2023Segment Anything (SAM)SAM combines reusable image features with point, box or mask prompts to predict candidate masks.Sources & context

    Contributors: Alexander Kirillov and colleagues

    What changed: Separates image encoding from changing region requests, including alternative interpretations of ambiguous prompts.

Follow how geometric constraints, learned recognition, motion and spatial outputs developed alongside one another, eventually adding language and prompts as inputs. Milestone spacing is not to scale.

These advances changed how features were obtained and predictions requested. Geometry still constrains physical measurements; spatial features support precise boundaries; temporal representations support motion and identity. A useful system can combine these contributions.

Visual representations

Local features and correspondence

A feature is a visual measurement useful for subsequent computation. A local interest point, also called a keypoint in feature matching, selects an image location; a descriptor summarizes its neighborhood numerically. Selection, description and matching have different jobs. A uniform patch provides little location information, while an edge constrains displacement mainly across the edge. Variation in several directions can better constrain a local match. These are examples of representations serving a task.

SIFT, the Scale-Invariant Feature Transform, describes a neighborhood after estimating its scale and orientation. Local brightness gradients contribute to orientation histograms; normalization reduces sensitivity to uniform contrast changes. Lowe's descriptor uses 128 components. Its encoding rule is prescribed rather than fitted through a task loss. Scale and orientation estimates can still fail, and viewpoint changes or occlusion can alter the visible neighborhood.

Appearance candidates and geometric agreement

Example

A candidate can resemble a feature yet violate the shared transformation.

Source locations

Three reference locations before translation.

02550751000255075100x (pixels)y (pixels)ABCABC
  • 1. A
  • 2. B
  • 3. C
Read coordinates and regions as data

X: 0–100 pixels; Y: 0–100 pixels, increasing down. Equal scale on both axes.

A (points)

(20, 20)

B (points)

(60, 20)

C (points)

(20, 60)

A: (24, 16)

B: (64, 16)

C: (24, 56)

Predictions and candidate

All correct correspondences use the same translation; C* is an alternative appearance match.

02550751000255075100x (pixels)y (pixels)ABCCandidate C*Candidate residualABCC*
  • 1. A
  • 2. B
  • 3. C
  • 4. Candidate C*
  • 5. Candidate residual
Read coordinates and regions as data

X: 0–100 pixels; Y: 0–100 pixels, increasing down. Equal scale on both axes.

A (points)

(40, 30)

B (points)

(80, 30)

C (points)

(40, 70)

Candidate C* (points)

(65, 55)

Candidate residual (polyline)

(40, 70); (65, 55)

A: (44, 26)

B: (84, 26)

C: (36, 77)

C*: (69, 52)

Constructed translation: x increases by 20 pixels and y by 10. A, B and C retain their identities. Candidate C* disagrees with C's predicted position; the orange segment is its residual.

Similar descriptors supply candidate correspondences, not established identities. RANSAC, introduced by Fischler and Bolles in 1981, repeatedly fits a geometric model from a small random sample and counts observations agreeing within a tolerance. A promising model can then be refitted using its supporting observations. In the figure's constructed translation, three matches agree on the same displacement while another candidate does not. The result depends on the chosen geometry, tolerance and sampling budget.

Convolutional feature hierarchies

An encoder transforms an image into features. A feature map keeps vectors arranged on a spatial grid; its channels are different learned measurements at each location. A reusable encoder is often called a backbone. A task head converts its features into categories, boxes or other predictions. Training fits numerical weights from examples; Machine Learning Fundamentals explains that process.

A convolutional neural network (CNN) builds features with local filters. Each convolution slides the same learned weights across neighborhoods, combining pixel or feature values across positions and channels. Padding adds border values; stride controls the spacing between output locations. Pooling summarizes neighborhoods, commonly by their maximum. Successive layers combine larger neighborhoods—their receptive fields—while downsampling makes the output grid coarser.

Equivariance means that moving the input moves its feature map correspondingly. Invariance means that the output stays unchanged. Shared convolution supports translation equivariance under compatible shifts and boundary handling; stride can break it for one-pixel shifts. Pooling can reduce sensitivity to precise location, which helps recognition but can remove distinctions needed for localization. More contextual coverage does not create new image evidence.

One filter, repeated across the image

Input and filter window
0000000
0000000
0099900
0099900
0099900
0000000
0000000
Filter weights
0.1110.1110.111
0.1110.1110.111
0.1110.1110.111
Output · 5 × 5
Cell 1 of 25

Output (1, 1): (0 × 0.111) + (0 × 0.111) + (0 × 0.111) + (0 × 0.111) + (0 × 0.111) + (0 × 0.111) + (0 × 0.111) + (0 × 0.111) + (9 × 0.111) = 1

Slide each filter over the bright rectangle: averaging blends neighborhoods, while directional differences respond to its edges. Change stride to space outputs farther apart and zero padding to include borders. These illustrative weights perform single-channel cross-correlation without flipping the kernel; no training, bias or activation is applied. Values are displayed to three decimal places; calculations use full precision. With reduced motion enabled, use Previous, Next, or select an output cell.

Learning through depth

LeNet-5 combined local convolutions, subsampling and classification; its original subsampling units included learned scale and bias. AlexNet extended learned recognition to a much larger natural-image task. Its winning 2012 competition submission achieved 15.3% top-five test error, versus 26.2% for the runner-up: the reference category was missing from all five proposed categories at those rates. That result averaged seven networks, including two with additional pretraining, rather than comparing one unchanged network under identical training conditions.

y=F(x)+xy=F(x)+x A residual block adds its learned transformation F(x)F(x) to a compatible shortcut input xx. It learns a change relative to the input rather than having to reconstruct the entire output through the transformed branch.

ResNet addressed higher training error in deeper plain networks. In its matched-depth ImageNet comparison, a 34-layer residual network had 25.03% top-one validation error versus 28.54% for the plain network under ten-crop evaluation. The shortcut improved optimization in that comparison; it does not restore detail lost through earlier downsampling.

Patches and spatial interaction

A patch is a small image region. ViT, the Vision Transformer, converts each patch's pixel values into a learned vector, or patch token, and adds position information. Attention lets tokens combine information from other tokens; Transformers and Attention explains how. A separate classification state summarizes the image for category prediction, while patch states retain their association with image locations.

N=HWP2N=\frac{HW}{P^2} For image height HH, width WW, and square patch width PP, divisible dimensions yield NN patches. Doubling both image dimensions quadruples NN and multiplies full attention's pairwise entries by sixteen. This counts attention relationships, not total runtime.

Context and spatial readout

Contextualization and spatial reduction are separate operations.

Image patches
Position information
Positioned vectors
Contextual states
Image summary
Spatial readout
Positioned patch representations acquire context. A summary serves image-level prediction; spatial readouts retain patch locations.
Read the diagram as text
  • Image patches.
  • Position information.
  • Positioned vectors.
  • Contextual states.
  • Image summary.
  • Spatial readout.
  • Image patches → Positioned vectors: Project.
  • Position information → Positioned vectors: Add.
  • Positioned vectors → Contextual states: Contextualize.
  • Contextual states → Image summary: Read classification state.
  • Contextual states → Spatial readout: Read patch states.

Global interaction is one design choice. Liu and colleagues' 2021 Swin Transformer computes attention within local windows, then shifts their partitioning so information crosses previous boundaries. Patch merging progressively reduces spatial resolution and increases feature width. This yields several feature scales for detection and segmentation. Fixed windows change the scaling relationship, but do not establish a universal latency advantage over convolution.

Architecture and training work together. Local structure can be built into the network or encouraged by pretraining tasks. How Transformers Finally Ate Vision examines this tradeoff through reconstruction and reusable features. The practical decision is which spatial information the resulting features expose to the intended readout.

Categories, objects and regions

Categories and visual similarity

A classification head converts features into logits, numerical scores before probability normalization. For a single-label task, softmax distributes probability across mutually exclusive categories. Training compares that distribution with the target category. A multilabel task instead permits several positive labels; separate sigmoid outputs and binary losses can represent them without forcing the scores to sum to one.

CLIP, Contrastive Language–Image Pretraining, learns coordinated image and text encoders from matching and mismatching pairs. Candidate category descriptions can then be compared with an image without fitting a new target-specific classification head. The image-text alignment chapter explains the shared space. Candidate wording and coverage matter: similarity ranks descriptions; it is not a calibrated probability that a claim is true.

The Paint.wtf drawing game applied this mechanism by ranking drawings against a text prompt using cosine similarity. Users discovered that writing the requested scene could score well. That shortcut exposed the difference between matching CLIP's representation and judging the intended drawing task. Likewise, naming a watch requires different visual distinctions from locating its hands. Retrieval infrastructure belongs in Search and Retrieval; here the essential choice is what kind of resemblance the features support.

Open-vocabulary recognition accepts category descriptions beyond a fixed numerical label list. Open-set recognition additionally allows rejection of unfamiliar categories. A closed-set classifier always has a highest-scoring known category, even for an unrelated input. The OpenMax study showed that low-confidence rejection alone did not reliably detect unknown inputs in its experiments. Flexible vocabulary and reliable rejection are separate capabilities.

Finding objects at different scales

Detection must allocate predictions to a variable number of object instances. A proposal is an intermediate candidate region. Ren and colleagues' 2015 Faster R-CNN shares convolutional features between a proposal network and a detector. Reference rectangles called anchors provide scales and aspect ratios; the proposal network predicts object-versus-background scores and box adjustments. The detector then processes those proposals.

The original 2015 YOLO instead predicts boxes and category information directly from an image grid in one network evaluation. The cell containing an object's center takes responsibility for it. Its fixed per-cell prediction scheme made precise localization of small objects difficult. These details describe the original model; later systems sharing the YOLO name need their own output specifications.

Context reaches a finer grid

Lateral features contribute detail that enlargement alone cannot supply.

Fine backbone mapInput-relative stride 16.
Coarser backbone mapInput-relative stride 32.
Fine lateral features
Projected coarse features
Upsampled coarse features
New fine pyramid level
New coarse pyramid level
A two-level FPN excerpt. Coarse projected features are enlarged and added to fine lateral features. Each resulting prediction level is a new representation.
Read the diagram as text
  • Fine backbone map. Input-relative stride 16.
  • Coarser backbone map. Input-relative stride 32.
  • Fine lateral features.
  • Projected coarse features.
  • Upsampled coarse features.
  • New fine pyramid level.
  • New coarse pyramid level.
  • Fine backbone map → Coarser backbone map: Backbone processing.
  • Fine backbone map → Fine lateral features: 1×1 projection.
  • Coarser backbone map → Projected coarse features: 1×1 projection.
  • Projected coarse features → Upsampled coarse features: Upsample ×2.
  • Fine lateral features → New fine pyramid level: Add, then 3×3 convolution.
  • Upsampled coarse features → New fine pyramid level: Add, then 3×3 convolution.
  • Projected coarse features → New coarse pyramid level: 3×3 convolution.

Object scale also affects which features are useful. Lin and colleagues' Feature Pyramid Networks, first posted in 2016, combine coarse contextual features with finer backbone maps. Lateral projections meet an upsampled top-down representation; further convolution produces new pyramid features. Prediction heads use multiple resolutions. Upsampling alone cannot recover discarded detail—the lateral features contribute information still available on finer grids.

Assignment and duplicate predictions

Several predictions may describe one object. Intersection over union, or IoU, measures shared region area divided by union area. Non-maximum suppression, or NMS, retains high-scoring boxes and removes lower-scoring boxes whose overlap exceeds a threshold. It compares predictions with predictions. Distinct overlapping objects can therefore be suppressed; class handling must also be specified.

Carion and colleagues' 2020 DETR approaches duplication through set prediction. A fixed collection of learned object queries produces class scores and boxes. During training, Hungarian matching finds a minimum-cost one-to-one assignment between predictions and labeled objects. Matched predictions receive class and box losses; unmatched predictions learn a no-object category. The box objective combines coordinate error with generalized IoU.

The two operations use different information. NMS filters candidates during inference without labels. DETR's assignment uses labels to define training supervision. Original DETR omits an NMS postprocessing stage, but one-to-one supervision does not guarantee perfect inference. Evaluation later performs another prediction-to-reference matching operation, whose purpose is scoring rather than training.

Dense labels and instance boundaries

Boxes include surrounding pixels, so they cannot directly specify a precise cutout or defect contour. Semantic masks assign categories but do not distinguish touching objects of the same category. Instance masks separate them. Panoptic segmentation combines individual countable “things,” such as cars, with “stuff,” such as sky. Each pixel receives one category and, where meaningful, an instance identifier. The panoptic task definition concerns visible regions, with no overlapping pixel assignments.

Dense prediction must preserve or recover spatial organization. Fully Convolutional Networks, or FCNs, reinterpret classification layers as convolutions to produce coarse category maps. Their upsampling and feature connections combine broad semantic information with finer appearance features. Whole-image processing shares computation across regions instead of independently classifying every patch.

Illustrative masks for a synthetic scene. Semantic labels group both oranges into one category; instance IDs distinguish the individual fruit. The panoptic view also labels the visible tabletop.

Ronneberger, Fischer and Brox's 2015 U-Net makes that combination explicit. A contracting path reduces resolution while increasing feature channels. An expanding path upsamples and concatenates aligned features from corresponding contracting levels, then predicts pixel classes. These skip connections supply localized information alongside coarse context. Original U-Net crops skip features because unpadded convolutions shrink their spatial support; its output covers an interior region. Weighted pixel supervision emphasizes, among other errors, boundaries between touching cells. Neither the skip paths nor enlargement alone guarantees fine contours.

He, Gkioxari, Dollár and Girshick's 2017 Mask R-CNN adds a mask branch for each proposed object. RoIAlign, region-of-interest alignment, samples features with interpolation instead of rounding region coordinates. This preserves alignment needed for pixel predictions. Classification and mask prediction remain separate: recognizing the right object does not establish the right contour.

The consequence is visible in portrait editing. Google's Magic Editor account describes a portrait segmentation model that missed fine hair strands, followed by image-processing refinement. A mask adequate for finding the subject could still produce conspicuous errors when used for blur or compositing.

Promptable segmentation

Segment Anything, or SAM, separates image encoding from region selection. An image encoder produces reusable features; a prompt encoder represents points, boxes or masks; a decoder predicts candidate masks. One point can refer to a whole object or a part, so the original system produces multiple candidates with estimated quality scores. Those scores predict overlap quality rather than measure it against a reference. Fine structures and boundaries can still be missed.

Prompt meaning can change the task. The SAM 3 team's 2025 concept-segmentation work lets a short noun phrase or image exemplar specify a concept whose matching instances should be found. A box around one dog can request all dogs; an instance-selection prompt requests that particular dog. Positive and negative exemplars refine the concept, while clicks can refine individual masks. Its simple-phrase contract does not imply arbitrary relationship reasoning. A system must know whether the user is naming a category, selecting an instance or correcting a boundary.

Reusable image, changing selection

A new prompt can reuse image features.

Image
Reusable image features
Point, box or mask prompt
Prompt representation
Mask decoder
Candidate masks
Image and prompt encoders supply different inputs to mask prediction. Candidate masks represent possible interpretations.
Read the diagram as text
  • Image.
  • Reusable image features.
  • Point, box or mask prompt.
  • Prompt representation.
  • Mask decoder.
  • Candidate masks.
  • Image → Reusable image features: Image encoding.
  • Point, box or mask prompt → Prompt representation: Prompt encoding.
  • Reusable image features → Mask decoder: Visual information.
  • Prompt representation → Mask decoder: Selection information.
  • Mask decoder → Candidate masks: Predict.

Space and visual language

From pixels to scene geometry

Spatial outputs need coordinate meanings. A semantic keypoint is a named landmark, such as a left wrist, rather than simply a repeatable texture location. A configuration of these landmarks describes articulated pose in image coordinates. Camera pose instead describes the camera's position and orientation relative to a coordinate frame. Neither a landmark's name nor its image coordinates supplies its depth.

Calibration estimates the camera model from known geometry. Intrinsic parameters describe focal scaling and the image's principal point; extrinsic parameters relate camera and world coordinates. A pixel constrains a viewing ray, leaving depth unresolved. Zhang's 1998 calibration report made calibration practical using a known planar pattern viewed at several orientations, with numerical refinement and radial-distortion estimation. Pattern accuracy, flatness and informative viewpoints still matter.

Two rays constrain depth

Example

One viewing ray admits several scene points.

Known camera separation

Camera centers are two meters apart in this planar example.

-2-101202468Lateral position (meters)Depth (meters)Camera centersLeft rayRight rayP: both raysQ: left ray onlyLeftRightPQ
  • 1. Camera centers
  • 2. Left ray
  • 3. Right ray
  • 4. P: both rays
  • 5. Q: left ray only
Read coordinates and regions as data

X: -2–2 meters; Y: 0–8 meters, increasing up. Equal scale on both axes.

Camera centers (points)

(-1, 0); (1, 0)

Left ray (polyline)

(-1, 0); (0.75, 7)

Right ray (polyline)

(1, 0); (-0.75, 7)

P: both rays (points)

(0, 4)

Q: left ray only (points)

(0.5, 6)

Left: (-1.1, 0.4)

Right: (1.1, 0.4)

P: (0.2, 3.8)

Q: (0.7, 6)

Constructed planar cross-section. P and Q share the left viewing ray; only P also lies on the chosen right ray. Physical scale assumes known camera geometry and a valid correspondence.

A second camera supplies another constraint if the same scene point can be matched. Epipolar geometry restricts its possible partner to a line in the second image, reducing the search space. Repeated textures and occlusion can still make the match wrong or unavailable. With usable calibrated correspondences, stereo triangulates depth.

Z=fBdZ=\frac{fB}{d} For rectified parallel cameras, ZZ is depth, ff focal length in pixels, BB the physical baseline and dd horizontal disparity in pixels under compatible coordinate conventions. Small disparity makes distant depth sensitive to matching error.

Learned monocular depth uses patterns acquired from examples to infer depth from one image. Depth Anything V2, introduced in 2024, predicts inverse depth with scale-and-shift ambiguity in its base models; metric depth requires separate adaptation. Its teacher learns from synthetic depth, labels real images, and supplies student targets. Correct near-versus-far ordering therefore answers a different question from accurate distances in meters.

Structure from motion estimates camera viewpoints and scene structure jointly from corresponding observations. Additional views constrain shared geometry, but moving objects and incorrect matches can violate that model, and physical scale is not automatic. Robotics develops how perception connects to state estimation and action.

Descriptions, referents and relations

A referring expression identifies an intended object through attributes or relationships: “the cup left of the bowl.” Recognizing cups and bowls does not select the referent. MAttNet, introduced in 2018, scores candidate regions using subject appearance, image location and neighboring-object relationships. It illustrates why reference resolution requires more than category matching.

Shilong Liu and colleagues' 2023 Grounding DINO integrates language into detection at several stages. Text and image features exchange information; language guides query selection; a decoder uses both representations to predict boxes and text associations. This supports category descriptions and referring-expression localization beyond a fixed class list. Its detector lineage named DINO is distinct from the self-supervised feature learner discussed later. Open-vocabulary detection still does not guarantee arbitrary relation reasoning or rejection of absent targets.

Spatial language also needs a reference convention. “Left” may mean image-left or a direction relative to an object. The CLEVR diagnostic dataset, posted in 2016 and published in 2017, pairs rendered scenes with executable operations for selecting attributes, following relations, counting and comparing. Its directional relations use an explicit camera-dependent convention. Supplying known scene information helps separate perception failures from failures to compose those operations.

In these constructed scenes, the same object categories are present but the answer changes: first the mug, then the tumbler. “Left” refers to image-left. Recognizing blue objects alone does not resolve the relationship.

The intended answer can also be empty. GRES, introduced in 2023, includes descriptions matching one, several or no objects. Its reported seated-person example recognized the person and bed yet selected someone sitting on a nearby chair. Recognizing the components had not established the stated relationship. These distinctions connect to reasoning across modalities; additional deliberation is developed in Reasoning and Test-Time Compute.

Visual features in language responses

A vision-language model uses visual information together with language inputs. A connector must make visual features usable by the language model; equal vector widths alone do not establish compatibility. In the LLaVA team's 2023 Visual Instruction Tuning, a learned linear projection maps CLIP grid features into the language model's embedding width. Those visual embeddings join instruction text as context for generating an answer.

Original LLaVA first trains the projection while freezing the vision encoder and language model, then updates the projection and language model on visual instructions while keeping the vision encoder frozen. This teaches both compatibility and response behavior. Alignment concerns learned correspondence; fusion concerns where information interacts. Neither guarantees that a generated statement accurately describes the image.

The intended answer boundary therefore matters. The Moondream account distinguishes extracting an expression from its image from solving the calculus problem it contains. That narrower contract guides training data and benchmarks. The speaker also warns that copying a stronger model's elaborate answers can encourage unsupported elaboration when the student cannot reproduce the knowledge or reasoning behind them. The relevant target is grounded task performance, not the appearance of a sophisticated answer.

Learning and adaptation

What visual supervision rewards

A training objective specifies which prediction errors fitting should reduce. The target determines what information the features must make accessible. Machine Learning Fundamentals explains objectives generally; visual targets differ particularly in how much spatial structure they supervise.

TargetRequired prediction
Category labelDistinguish the image's target category
Object boxesAssign categories and spatial extents to instances
Pixel labelsRecover spatial category membership
Matching image and descriptionPrefer the paired description to alternatives

Self-supervised learning derives targets from observations rather than requiring a person to label every prediction. Chen and colleagues' 2020 SimCLR independently transforms an image into two views, encodes them, and compares projections of their representations against views of other images. Cropping, color changes and blur define which changes the training task encourages the representation to tolerate. The projection head used for this objective is discarded for downstream use. Contrastive learning explains the general match-versus-alternative mechanism.

A transformation encodes an assumption about meaning. Color distortion may be useful for recognizing a category while damaging a task whose answer depends on color. A crop can preserve an image's broad subject while removing the small region needed for its label. More transformed examples help only when the resulting targets remain valid.

Reconstruction and self-distillation

Masked prediction creates another learning problem. He and colleagues' Masked Autoencoders, posted in 2021 and published in 2022, send only visible patches through the encoder. A smaller decoder combines their representations with positioned mask tokens and predicts patch pixels. Reconstruction loss is measured on the masked patches. The encoder must learn useful context, while the decoder is discarded for downstream recognition. A plausible reconstruction need not reproduce the actual hidden content.

The encoder sees only visible patches. Mask tokens join before decoding; training compares reconstructed pixels with the withheld regions. The bicycle and reconstruction are illustrative, not outputs from a trained MAE.

Self-distillation instead learns from another evolving model's outputs. Caron and colleagues' 2021 DINO trains a student to match teacher distributions across image views, including smaller student crops. The teacher follows an exponential moving average of student parameters. Centering and sharpening its outputs counter different collapse tendencies, where predictions cease to distinguish images. These targets are learned distributions, not supplied human category labels. Object-related attention patterns appeared in its experiments, but recognizable patterns alone do not establish precise segmentation.

DINOv2, introduced in 2023, combines image-level agreement with masked patch targets from an unmasked teacher view. These are learned feature distributions rather than reconstructed pixels. Frozen-encoder experiments demonstrate readouts for classification, segmentation and depth; downstream labels are still needed to train those readouts.

The 2025 DINOv3 report shows why global and local utility need separate attention. During its extended pretraining experiments, image-classification performance improved while segmentation from patch features deteriorated. Gram anchoring encouraged pairwise patch-feature similarities to resemble those of an earlier teacher with stronger dense features. It preserved local relationships without fixing every feature vector in place. The lesson concerns the measured training configurations: improving an image summary can coexist with weakening a spatial readout.

Adapting reusable features

Adaptation determines what can change for the new task. A frozen encoder retains its fitted weights. A linear probe trains only a weighted-sum readout; a richer head can expose spatial outputs. Fine-tuning updates the encoder as well. Readout evaluation explains what a probe establishes, while Post-training develops parameter adaptation. Prompting changes supplied instructions or examples without changing weights.

There is no guaranteed ordering from probing to better fine-tuning. A 2022 study found configurations where full fine-tuning improved in-distribution accuracy while reducing performance under distribution shift relative to probing. Updating useful features can change relationships that previously transferred. Compare adaptation methods on both the intended domain and the conditions whose performance must be retained.

Target meaning can be as important as parameter count. RF100-VL, introduced in 2025, supplies reviewed descriptions and visual examples for challenging domain-specific classes. Its ten-shot subsets contain ten object instances per class, not necessarily ten images. Prompting a frozen model and updating parameters are separate evaluation regimes. The Vision AI in 2025 discussion illustrates why a small adapted detector is a useful challenge to zero-shot deployment, rather than assuming the general model will win.

Preserving annotation meaning

Labels specify an interpretation of the observation. Visible segmentation marks observed extent; amodal segmentation estimates a complete object including hidden portions. Those hidden boundaries are inferred targets. Missing labels also differ from confirmed absence. LVIS records verified presence, verified absence and unassessed categories, with a separate indication of whether all instances were annotated. Discarding those distinctions can turn valid predictions into apparent errors.

Class imbalance means some labels occur much more often than others. A model can perform well overall while missing rare defects. Resampling changes which examples training encounters; loss weights change their contribution to fitting. Neither justifies artificially balancing the deployment evaluation or assuming the resulting scores are calibrated. Negative images and difficult normal appearances belong in the task definition, alongside examples of the rare target.

Clipping changes recoverable extent

Example

Inverse coordinates recover only surviving extent.

Original and crop

Dashed orange boundary selects the crop; green marks surviving box extent.

02550751000255075100x (pixels)y (pixels)Surviving extentCanvas boundaryCrop boundaryOriginal boxClipped stripCrop
  • 1. Canvas boundary
  • 2. Crop boundary
  • 3. Original box
  • 4. Surviving extent
Read coordinates and regions as data

X: 0–100 pixels; Y: 0–100 pixels, increasing down. Equal scale on both axes.

Canvas boundary (polyline)

(0, 0); (100, 0); (100, 80); (0, 80); (0, 0)

Crop boundary (polyline)

(20, 10); (80, 10); (80, 70); (20, 70); (20, 10)

Original box (polyline)

(10, 20); (50, 20); (50, 50); (10, 50); (10, 20)

Surviving extent (polygon)

(20, 20); (50, 20); (50, 50); (20, 50)

Clipped strip: (12, 57)

Crop: (55, 16)

Resized and padded

The new canvas is 40×40 pixels. The same scale is retained for comparison.

02550751000255075100x (pixels)y (pixels)Surviving extentCanvas boundaryCrop boundarySurviving box only
  • 1. Canvas boundary
  • 2. Crop boundary
  • 3. Surviving extent
Read coordinates and regions as data

X: 0–100 pixels; Y: 0–100 pixels, increasing down. Equal scale on both axes.

Canvas boundary (polyline)

(0, 0); (40, 0); (40, 40); (0, 40); (0, 0)

Crop boundary (polyline)

(5, 5); (35, 5); (35, 35); (5, 35); (5, 5)

Surviving extent (polygon)

(5, 10); (20, 10); (20, 25); (5, 25)

Surviving box only: (4, 48)

Crop [20,10,80,70], resize to 30×30, then pad five pixels per side. Box [10,20,50,50] clips to [20,20,50,50], becoming [5,10,20,25]. Inverting that box cannot recover the discarded left edge.

Coordinates and target validity

Geometric augmentation must transform images and spatial targets together. Under continuous box-edge coordinates, a box's area is width times height, without adding one for inclusive integer endpoints. Cropping subtracts the crop origin; resizing multiplies coordinates; padding adds an offset.

x′=sx(x−l)+px,y′=sy(y−t)+pyx'=s_x(x-l)+p_x,\qquad y'=s_y(y-t)+p_y Here (l,t)(l,t) is the crop origin, sx,sys_x,s_y use the actual resized dimensions, and px,pyp_x,p_y are left/top padding. Invert surviving coordinates by subtracting padding, dividing by scale and restoring the origin. Clipped extent cannot be recovered by inversion.

Categorical masks need different numerical treatment from image intensities. Torchvision's segmentation transforms share crop and flip choices, resize targets with nearest-neighbor interpolation, and normalize only the image. Nearest-neighbor preserves selected label values, but downsampling can still erase a thin region. Even perfect coordinate updates do not validate the label: a crop that removes the defect may no longer support the original defective-image category.

Named landmarks

Named landmarks add identity and annotation eligibility. COCO body-keypoint conventions use visibility 0 for unannotated, 1 for annotated but not visible, and 2 for annotated and visible. Hidden does not therefore mean unannotated. Detectron2 uses continuous coordinates and adds 0.5 when loading COCO integer pixel indices.

Detectron2 transforms landmark positions with the image, marks out-of-bounds points unannotated and zeros their coordinates. Surviving points retain visibility. An odd number of horizontal reflections also permutes left/right landmark records. The zero coordinates are a storage convention for unavailable annotations, not an observation at the image origin.

TransformationAdditional target operation
Crop excludes a wristMark unannotated; zero coordinates
Horizontal reflectionExchange left/right landmark records
Point survives unchanged eligibilityRetain its visibility value

These rules should be part of the annotation specification, then checked through annotation review. Spatial alignment, semantic identity and eligibility are separate properties to preserve.

Time, motion and identity

Frames, sampling and temporal order

A frame is an image associated with a time; a clip is a selected temporal interval. A video representation retains features across those observations. Frame index alone does not establish elapsed time. FFmpeg's frame representation distinguishes presentation timestamps from decoding-related timing, and presentation time is not necessarily wall-clock capture time.

Temporal sampling selects which moments reach the model. In the constructed figure, an event lasts from 1.1 to 1.7 seconds. Samples at integer seconds miss it entirely; adding the 1.5-second observation captures part of it. That observation establishes presence at one moment, not the exact boundaries or complete action. Clip windows impose another limit: a relationship between distant events needs both events to remain accessible.

A brief event between samples

Example

Changing observation times changes event coverage.

One-second spacing

Every selected observation falls outside the event.

00.81.62.43.2-0.20.1750.550.9251.3Video time (seconds)Event active (dimensionless)Event stateSelected observations
  • 1. Event state
  • 2. Selected observations
Read coordinates and regions as data

X: 0–3.2 seconds; Y: -0.2–1.3 dimensionless, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Event state (polyline)

(0, 0); (1.1, 0); (1.1, 1); (1.7, 1); (1.7, 0); (3, 0)

Selected observations (points)

(0, 0); (1, 0); (2, 0); (3, 0)

Half-second spacing

The event is unchanged; one added observation falls inside it.

00.81.62.43.2-0.20.1750.550.9251.3Video time (seconds)Event active (dimensionless)Event stateSelected observationsObserved active
  • 1. Event state
  • 2. Selected observations
Read coordinates and regions as data

X: 0–3.2 seconds; Y: -0.2–1.3 dimensionless, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Event state (polyline)

(0, 0); (1.1, 0); (1.1, 1); (1.7, 1); (1.7, 0); (3, 0)

Selected observations (points)

(0, 0); (0.5, 0); (1, 0); (1.5, 1); (2, 0); (2.5, 0); (3, 0)

Observed active: (1.5, 1.16)

Constructed event active from 1.1 to 1.7 seconds. Integer-second samples all miss it. Half-second samples include one active observation, which still does not establish exact event boundaries.

Retaining frames is still different from retaining order. Averaging the same frame vectors yields the same result when their order is reversed. Yet opening and closing can contain the same states in opposite sequences. The 2019 Retro-Actions study treats transformations as potentially preserving a class, exchanging it with another, or creating an invalid action. Reversal may turn covering into uncovering, while other reversals imply implausible physics. Temporal augmentation therefore needs a semantic rule, just as image augmentation does.

Learning across space and time

A temporal model allows observations at one time to affect the interpretation of another before they are summarized. Different architectures supply this interaction in different ways. Optical flow, developed next, estimates apparent image displacement and can provide an explicit motion input.

The two-stream model separates an RGB appearance network from a network processing stacked horizontal and vertical flow fields. Combining their predictions lets motion complement objects and scene appearance. Image pretraining helps the appearance path, while the explicit flow computation contributes motion structure. The original action-classification results do not supply event boundaries or persistent identities.

Carreira and Zisserman's 2017 I3D, an inflated three-dimensional convolutional network, extends image filters across time. Repeating and scaling a pretrained spatial filter initializes a temporal filter with the same response to a repeated still image. Subsequent video training learns space-time features. This transfers an image-network starting point; inflation itself does not mean motion has already been learned.

Bertasius, Wang and Torresani's 2021 TimeSformer represents video through patches indexed by position and frame. Its divided-attention block first exchanges information across time at matching patch positions, then across positions within each frame. This avoids one joint attention operation over every space-time pair. Matching coordinates across frames is an architectural correspondence rule; a moving object may occupy different coordinates, so this is not object tracking.

Apparent motion

Optical flow represents apparent displacement with two components at each image location, measured in pixels over a specified frame interval. Brightness constancy assumes that a moving pattern preserves its intensity. A straight edge constrains motion perpendicular to itself but leaves parallel motion unresolved: the aperture problem. Additional neighborhood assumptions are needed to choose a complete vector.

Horn and Schunck add a preference for smoothly varying flow. Their 1981 work also demonstrates that physical motion can occur without corresponding brightness-pattern movement, as with a uniformly shaded rotating sphere. Occluding boundaries produced large errors. Camera movement, changing illumination and object movement can all change images in different ways, so flow is not directly calibrated physical velocity.

One edge, several possible motions

Example

Tangential motion remains unconstrained.

Local edge observations

Only the edge's normal displacement is determined.

0246802468x (pixels)y (pixels)Earlier edgeLater edgePossible displacement APossible displacement BPossible displacement C
  • 1. Earlier edge
  • 2. Later edge
  • 3. Possible displacement A
  • 4. Possible displacement B
  • 5. Possible displacement C
Read coordinates and regions as data

X: 0–8 pixels; Y: 0–8 pixels, increasing down. Equal scale on both axes.

Earlier edge (polyline)

(3, 0.5); (3, 7.5)

Later edge (polyline)

(5, 0.5); (5, 7.5)

Possible displacement A (polyline)

(3, 4); (5, 2)

Possible displacement B (polyline)

(3, 4); (5, 4)

Possible displacement C (polyline)

(3, 4); (5, 6)

A vertical edge shifts two pixels right. All three arrows share that perpendicular displacement but differ along the edge. This local construction does not identify physical motion.

Lucas and Kanade's 1981 registration method instead estimates shared alignment over a local region through iterative gradient-based updates. Coarse-to-fine processing can accommodate larger initial displacement, trading fine detail for a broader capture range. Both approaches add assumptions to incomplete observations. Neither supplies a persistent object identity.

Maintaining object identity

A track is a maintained hypothesis that observations belong to one object. Association connects new detections to existing tracks. An identity switch assigns observations inconsistently across objects. Accurate per-frame boxes can coexist with these errors because finding an object and maintaining its identity are different problems.

Bewley and colleagues' 2016 SORT predicts box motion with a Kalman filter, which maintains an estimate and uncertainty, then uses overlap costs and one-to-one Hungarian assignment to associate detections. Tracks are updated, created or removed according to observations and lifecycle rules. Its simple motion model and lack of long-term re-identification make prolonged occlusion difficult.

A track outlives an observation

Example

Retained identity is a hypothesis.

1 / 3 · Observed

A detection initializes T.

New this stepEarlier detectionAn observation, not the physical object.
New this stepTrack TRetained identity hypothesis.
Constructed association sequence. A track persists through a missed observation; two later candidates leave its identity unresolved.
Read the diagram as text
  • Earlier detection. An observation, not the physical object.
  • Track T. Retained identity hypothesis.
  • Missed observation.
  • Candidate A.
  • Candidate B.
  • Association unresolved.
  • Earlier detection → Track T: Initializes.
  • Track T → Missed observation: Persists through.
  • Track T → Candidate A: Motion-compatible.
  • Track T → Candidate B: Appearance-compatible.
  • Candidate A → Association unresolved: Competing candidate.
  • Candidate B → Association unresolved: Competing candidate.
  1. Observed. A detection initializes T. Active: Earlier detection, Track T. New: Earlier detection, Track T.
  2. Temporarily unobserved. T remains available despite a missing detection. Active: Earlier detection, Track T, Missed observation. New: Missed observation.
  3. Competing reappearances. New candidates supply incomplete association evidence. Active: Earlier detection, Track T, Missed observation, Candidate A, Candidate B, Association unresolved. New: Candidate A, Candidate B, Association unresolved.

Wojke, Bewley and Paulus's 2017 Deep SORT adds learned appearance descriptors and prioritizes recently observed tracks during matching. Appearance helps when motion alone is ambiguous, but similar appearance does not prove identity. Its appearance model was trained for person re-identification; transfer to another object domain needs assessment.

Mask propagation uses another form of temporal state. SAM 2, introduced in 2024, conditions frame features on stored predictions and prompted frames. Later prompts can correct propagated masks, and a presence prediction allows the target to be absent. An offline correction can use a prompt from later in the recording; that result should not be interpreted as causal streaming performance. Stored state helps continuity but does not remove uncertainty when an object disappears and a similar one returns.

Actions, events and video language

Actions and visible state changes

A trimmed clip is selected around an action; classifying it assumes that useful temporal extent is already supplied. An untrimmed recording includes unknown boundaries and unrelated activity. Temporal localization predicts event intervals. Identifying participating objects adds a spatial requirement. Online action detection uses only observations available up to the current frame; offline analysis can use later context.

SlowFast, posted in 2018 and published in 2019, separates slowly changing appearance from faster motion through connected pathways. One published configuration samples four and 32 frames from the same 64-frame clip. The fast pathway uses fewer channels while retaining greater temporal resolution until pooling. The contribution is a division of representational work; those sample counts do not guarantee capture of every brief event.

Krishna and colleagues' 2017 dense-captioning work asks for multiple, possibly overlapping event intervals and a description of each. The Ego4D consortium's 2022 tasks further separate whether an object changed, when the change occurred and which object changed. Precondition, change and postcondition annotations connect actions to visible consequences. Different motions can produce similar changes; the visible sequence alone does not establish intention or a complete causal account.

An event's location in video time is separate from when a consumer receives the result. Shou and colleagues' 2018 action-start task predicts a class and onset time from sequential observations. A correct predicted onset does not measure elapsed alert-delivery delay. A live application needs explicit start and delivery boundaries, as explained in latency measurement.

Answers grounded in video

Video-text retrieval returns matching clips or intervals. Event captioning produces descriptions, while video question answering returns a response to a particular information need. A broad summary can identify the activity yet omit the brief event needed to answer a question about order. Temporal grounding must preserve the interval supporting that answer.

Hu Xu and colleagues' 2021 VideoCLIP learns compatible representations from instructional videos and temporally overlapping transcripts, using similar videos as difficult alternatives. Pooled representations support clip matching; token-level representations support finer action and step tasks. Its question-answering evaluation selects among candidate answers rather than generating unrestricted responses. Imperfect narration remains an imperfect training target.

Bo Li and colleagues' 2024 LLaVA-OneVision encodes individual frames, projects their features into a language model's input space and reduces video feature-token counts through interpolation. This makes room for more frames, exchanging spatial detail per frame for temporal coverage. More frames do not automatically preserve small objects, cross-frame identity or precise event boundaries. The choice must follow the question: a tiny indicator and a long event sequence impose different input demands.

For an order-dependent answer, retain the relevant observations with their timestamps and assess both the response and its supporting intervals. A source link makes inspection possible; it does not validate the claim. Multimodal alignment and fusion explain cross-signal learning, while Search and Retrieval covers indexing and ranking the resulting units.

Ambiguity and changing conditions

Missing information and uncertain interpretations

An uncertain answer can have different causes. The observation may omit the required information, the target definition may allow several interpretations, or the model may fail to use visible evidence. A learned prior can supply a plausible hidden shape without making that shape observed. Predictive uncertainty is therefore inseparable from which inputs and target definitions are available.

LimitationPotentially useful interventionWhat remains unresolved
Detail lost only during whole-image reductionInspect a crop from the retained originalWhether the source detail is sufficient
Surface hidden by another objectAcquire a revealing viewpointAny still-occluded extent
Unclear whole-object versus part targetClarify the requested regionPrediction accuracy after clarification
Event not yet completedObserve later frames where permittedWhether the application can wait

Unknown-category rejection is another task, distinct from low confidence among known labels. Deferral can limit accepted errors, but it transfers work to another process. Coverage is the accepted fraction; selective risk is error among accepted cases. Thresholds require representative labeled validation, a defined loss and enough accepted observations. Confidence alone supplies none of those guarantees. Deferral and review capacity explains how to assess the resulting human workload.

Capture changes and visual shortcuts

Distribution shift changes the conditions under which performance was established. In vision, a new camera can change illumination, viewpoint, background and category frequencies simultaneously. The general shift framework helps name these changes, but the visual diagnosis must identify which measurements or learned cues stopped transferring.

A shortcut is a predictive cue that works in the available data without satisfying the intended visual distinction. Geirhos and colleagues' 2018 preprint and 2019 study constructed images whose shape and transferred texture suggested different categories. Tested ImageNet CNNs favored texture more often than people; stylized-image training shifted the same ResNet-50 architecture toward shape. Training data changed cue preference without changing architecture.

Preference under deliberately conflicting cues is not identical to reliance in ordinary recognition. A later controlled-suppression study found tested CNNs more vulnerable to disrupted local shape than to texture suppression in its selected experiment. It also showed that purportedly texture-removing operations can damage shape. These results qualify the interpretation of “texture bias” without erasing the earlier cue-conflict finding.

Background-substitution experiments provide another diagnostic. Keeping the foreground while changing background class information can alter recognition; class-consistent replacement backgrounds help control for compositing artifacts. The experiment tests a particular cue, not its frequency in a factory or wildlife deployment.

ImageNet-C reserves algorithmic noise, blur, weather and digital corruptions for evaluation. It measures degradation under specified transformations. Naturally different camera locations, such as those in iWildCam, test a broader mixture of changes. Passing the former does not establish the latter. Even confidence calibration can deteriorate under stronger shift, as demonstrated in uncertainty-under-shift experiments. Remedies should follow the cause: improve capture, collect representative examples, use valid augmentation, adapt the model or narrow the operating conditions.

Measuring visual predictions

Scoring categories and detected objects

A metric must match the output claim. Image-level accuracy counts correctly classified images; per-class reporting exposes failures that common categories can hide. Detection additionally needs category-compatible spatial matches. Precision and recall distinguish correctness among predictions from coverage of references.

IoU⁡(A,B)=∣A∩B∣∣A∣+∣B∣−∣A∩B∣\operatorname{IoU}(A,B)=\frac{|A\cap B|}{|A|+|B|-|A\cap B|} Here A,BA,B are regions and vertical bars denote area. Two 10-by-10 boxes shifted five pixels horizontally share area 50 and have union area 150, giving IoU 1/31/3.

COCO ranks detections by score and matches eligible references per class and overlap threshold. Ordinary references match once; duplicates become false positives. Moving down the ranked list changes precision and recall. Average precision (AP) summarizes this curve using interpolated precision at 101 recall levels. Main COCO AP also averages classes and ten IoU thresholds from 0.50 to 0.95, with a 100-detection cap. AP50 uses only IoU 0.50. Crowd, ignored annotations and object-size subsets have additional rules.

Constructed score order with two reference objects:
PredictionResultPrecisionRecall
FirstMatches object A11/2
SecondDuplicates A1/21/2
ThirdMatches object B2/31

Protocols are part of the number. VOC 2007 used eleven-point interpolated AP, unlike COCO's recall grid. RF100-VL uses a 500-detection limit because some images contain more than 100 objects. A comparison must preserve these rules, adaptation conditions and category meanings; sharing a metric name is insufficient.

Scoring masks and spatial claims

A large correct background can hide a missed small foreground in pixel accuracy. For one foreground class, true positives (TP) are correctly predicted foreground pixels, false positives (FP) are extra foreground predictions, and false negatives (FN) are missed foreground pixels. IoU is TP/(TP+FP+FN); binary Dice is 2TP/(2TP+FP+FN). Both score foreground overlap without counting correctly predicted background. Dice's differentiable variants need their own definitions; an empty foreground also requires an explicit convention.

Aggregation changes the claim. Cityscapes accumulates a dataset confusion matrix before computing class IoUs, then averages valid classes equally. Ignored reference pixels are treated differently from evaluated pixels predicted as an ignored class. Averaging image-level IoUs or filling absent classes with zero produces another metric.

Boundary IoU, introduced in 2021, compares bands near predicted and reference contours. Its controlled dilation, erosion and displacement tests show how contour quality can differ from area agreement. Band width changes sensitivity, and errors far from both boundaries can receive little attention. Boundary and interior assessment are complementary. Instance separation and correct referent selection remain additional properties; an absent target should not be rewarded with a plausible-looking mask.

Other spatial outputs

Object Keypoint Similarity (OKS) scores corresponding annotated landmarks, scaling coordinate error by reference object area and landmark-specific tolerance. Annotated occluded landmarks participate; region overlap does not.

Depth assessment must state whether scale is independently recovered. The 2025 monocular-depth challenge fits a scale and shift against reference depth before scoring. Such errors measure agreement after reference-assisted alignment, not autonomous metric scale. Relative depth ordering, unaligned distances and reconstructed geometry answer different questions.

Relational claims also need direct tests. Winoground pairs images with captions containing the same words in different orders; group success requires all paired comparisons to be correct. This probes composition rather than region localization. CLEVR's executable count and relation operations offer another diagnostic: correct object categories alone do not establish the right count or relationship.

Scoring identities and events

Tracking evaluation must preserve identity through time. Ristani and colleagues' 2016 IDF1 uses global trajectory matching to count correctly identified detections, identity false positives and identity false negatives. A long wrong association can therefore matter differently from a brief switch. This offline matching scores the tracker; it is not the tracker's online association algorithm. HOTA, introduced in 2020, separates detection and association quality, exposing missed objects, extra detections, split trajectories and merged identities.

Temporal IoU replaces region area with interval duration. In the figure, reference [2,6] and prediction [4,8] share two seconds within a six-second union, giving 1/3. ActivityNet ranks predictions by score and matches same-class intervals within the same video, allowing each reference one match per threshold. Unmatched predictions are false positives. Its default summary averages AP across classes and ten temporal-overlap thresholds from 0.50 to 0.95. The comparison concerns predicted intervals, not delivery time.

Class agreement leaves timing unresolved

Example timings

Intervals and duplicate predictions require separate scoring.

Recording0–10 secondsDuration 10 seconds
Reference event2–6 secondsDuration 4 secondsWithin Recording
Shifted prediction4–8 secondsDuration 4 secondsWithin Recording
Duplicate prediction2.2–6.2 secondsDuration 4 secondsWithin Recording
All spans use video time. Reference and shifted prediction overlap for two seconds; their union is six seconds. The duplicate cannot create a second reference event. Overlapping durations must not be summed as elapsed recording time; these spans show no alert-delivery timing.
Read the diagram as text
  • Recording. 0 to 10 seconds; duration 10 seconds.
  • Reference event. 2 to 6 seconds; duration 4 seconds. Parent: Recording.
  • Shifted prediction. Same class. 4 to 8 seconds; duration 4 seconds. Parent: Recording.
  • Duplicate prediction. Another prediction for the same event. 2.2 to 6.2 seconds; duration 4 seconds. Parent: Recording.

Onsets, exposure and processing budgets

An onset error, an excess prediction lasting several seconds and unfinished video processing require different measurements. ActEV's Sequestered Data Leaderboard (SDL) protocols illustrate the latter two.
ProtocolWhat is matched or countedTiming meaning
Action-start detection, 2018A prediction must have the correct class and an onset within a chosen absolute offset from the annotated start. Each start can match once. AP is aggregated by class.Error in locating the onset on the video timeline, rather than time taken to deliver an alert.
ActEV SDL, 2019Confidence-sensitive, one-to-one matching uses label and temporal-overlap requirements. Miss probability is the fraction of reference instances left unmatched. On evaluated frames, excess predicted instances accumulate over time; this instance-time is divided by duration without the target activity.The selected frames define eligible exposure. Instance-time combines excess instance count with duration; it does not count separate alerts. Unmatched predictions can increase false-alarm time without directly increasing miss probability.
ActEV SDL, 2021 time-limited scoringDesignated processing calls are timed. Exceeding the real-time workload budget treats execution as stopped within the offending partition, incurring misses there and in subsequent partitions.Whether processing finishes within its budget, rather than elapsed time from an individual event to its delivered alert.

The 2019 NIST plan assesses predicted activity over evaluated video time. Its 2021 successor, reproduced here, also penalizes processing left unfinished under a workload budget. Neither records the interval from an individual event's occurrence to delivery of its alert, so that application requirement needs a separate measurement.

To measure false alerts per recording hour, define one alert, duplicate handling and eligible recording coverage. To measure delivered-alert delay, define the event or capture start boundary, delivery endpoint and relationship between their clocks. Report missing and late results explicitly; averaging only delivered successes can omit the failures that matter most. Latency boundaries and reporting rules develop these application measurements.

Video-language diagnostics need a further separation. TempCompass, introduced in 2024, constructs related videos with similar static content but changed speed, direction, attributes or event order. Matched single-frame, shuffled-frame and reduced-frame tests can investigate what temporal information a system uses. Failure to distinguish reversed order supports a targeted diagnosis; it does not estimate production failure frequency or establish that every error is temporal.

Usefulness in the intended setting

Testing the complete visual task

An independent test separates the relationships relevant to the intended claim. Neighboring frames, derived crops and repeated views of one object are different files but may carry nearly the same observation. Group by video, capture session, person, object or camera as the question requires. New-camera performance and later-day performance at an existing camera are different claims. Workload coverage and information boundaries explain how to specify them.

The iWildCam split revision is a concrete example. Its original in-distribution split could place images captured seconds apart by the same camera in different partitions. Dataset version 2.0 keeps camera-days together. The release also changed image membership and categories, so a score change across versions cannot be attributed solely to grouping. The broader WILDS task distinguishes same-camera days from unseen camera locations.

Test the complete input-to-output path. Confirm decoded orientation and channels, actual model input dimensions, inverse coordinate mapping and timestamp interpretation. Unavailable frames are missing coverage, not proof that no event occurred. Inspect annotation ambiguity and collection conditions as well as model errors. Controlled stress tests isolate sensitivities; new-site tests assess transfer; prospective operation observes the system under arriving workload.

Human corrections and review are part of the delivered task. A benchmark improvement can fail to help if it creates more work locating mistakes or correcting boundaries. Keep offline and live evidence distinct and measure the human workflow when claiming usefulness. The tested configuration includes acquisition, preprocessing, model, readout and review policy.

Choosing a practical vision system

Inspection

Industrial inspection begins with the distinction that changes disposition: a missing part, a crack, an incorrect assembly or a measurable contour. Capture control can be decisive. A 2023 automotive inspection study coordinates rotation, lighting and cameras, using configured image-processing checks and manual reinspection. Its initial 60-component test correctly flagged 15 defective components, falsely flagged four acceptable ones and correctly accepted 41. No misses were observed among those 15 defective cases; that finite result does not establish universal reliability.

A model can rank defective images above normal ones while locating the wrong regions. A 2025 forged-parts study obtained an image-level AUROC around 0.87. AUROC—the area under the receiver operating characteristic curve—summarizes discrimination as a score threshold varies; it does not assess defect location. The model's anomaly maps highlighted backgrounds and regions inconsistent with defects. Restricting evaluation to similar viewing angles improved results but simplified the tested workload. Inspection therefore needs assessment of the required region output under the camera conditions it will encounter, alongside image-level discrimination.

Mapping and area

Land cover describes surface categories such as trees, crops, water and built areas. Brown and colleagues' 2022 Dynamic World produces category-probability maps from individual Sentinel-2 acquisitions on a ten-meter grid. Per-acquisition predictions support selected time periods and change inspection. Cloud-masked locations are unavailable observations, and grid spacing does not guarantee equally accurate boundaries.

If the desired result is actual land area, counting classified pixels can inherit classification bias. Olofsson and colleagues' 2014 guidance uses probability sampling and higher-quality reference assessments for error-adjusted area estimates and uncertainty intervals. Samples stratified by mapped class must be weighted by the strata's area shares. A useful segmentation map and a defensible area estimate are related but different products.

Footage and interfaces

Footage retrieval needs a useful interval, sufficient temporal coverage and a way to inspect the result. A sampled-frame language interface can support flexible requests, while a specialized temporal predictor may fit a stable event vocabulary. Compare missed brief events, interval accuracy, processing time and actual review effort. The interface's convenience does not establish a reduction in review work.

RequirementUseful starting pointDecisive assessment
Stable appearance under controlled captureConfigured measurements or a specialized predictorMissed conditions and false reinspection
Accurate selectable regionsDense head or promptable segmentationBoundary quality and correction effort
Flexible visual requestsVision-language interfaceGrounded answers on the intended tasks
Events across a recordingTemporal representations with interval outputsCoverage, timing and identity where required

Annotation availability, output precision, device constraints and review capacity determine which starting point earns further development. Local and On-Device AI covers execution near the camera; Multimodal Models and Applications covers combining signals; Document Understanding and OCR covers document-specific structure; and Robotics connects perception to physical action.

Open questions

  1. Preserving fine visual distinctions while making features useful to language remains difficult. Global summaries and dense features can improve differently. Progress would combine language access with separately demonstrated localization and correspondence quality.

  2. Long-video systems must allocate finite representation capacity between temporal coverage and spatial detail. Missing a brief event and losing a small visual cue have different remedies. Progress would show controlled gains on both, with interval grounding and identity assessment rather than only aggregate answer scores.

  3. Combining domain descriptions, visual examples and parameter adaptation is still an empirical design problem. Extra inputs can fail to help if the model cannot integrate their meanings. Progress would isolate each input's contribution under equal adaptation conditions and measure transfer to unseen acquisition settings.

  4. Deferral remains hard when cameras and scene populations change. A threshold useful on familiar data can misrank failures after shift, while excessive review can erase the application's benefit. Progress would retain low accepted error and manageable review demand under representative new conditions.

  5. Model-assisted annotation needs comparisons that include correction effort and downstream utility. Generating many labels or obtaining agreement among models does not establish accurate targets or saved human work. Progress would measure total annotation and correction effort under equal labeling budgets, followed by independent task assessment.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

191 matching talks

TalkSpeakerEventYear
Suman DebnathAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer Summit 20252025
Ishan AnandAI Engineer World's Fair 20252025
Angelos PerivolaropoulosAI Engineer Europe 20262026
Abi AryanAI Engineer Summit 20232023
Joseph NelsonAI Engineer Summit 20232023
Evaling Video Slop

Transcript reviewed

Maor BrilAI Engineer World's Fair 20262026
Angel Ortmann LeeAI Engineer World's Fair 20262026
Doug GuthrieAI Engineer World's Fair 20252025
Building security around ML

Transcript reviewed

Dr. Andrew DavisAI Engineer World's Fair 20242024
Nader Khalil, Alex Cheema, Matthew Berman, Ahmad Osman, Joseph NelsonAI Engineer World's Fair 20262026
Gaurav MishraAI Engineer World's Fair 20262026
Why MLX

Transcript reviewed

AI Engineer Europe 20262026
Nikhil AbrahamAI Engineer World's Fair 20252025
Sander DielemanAI Engineer Europe 20262026
Dmitry PetrovAI Engineer World's Fair 20262026
Randall HuntAI Engineer World's Fair 20252025
Paige Bailey, Guillaume Vernade, Ian BallantyneAI Engineer Europe 20262026
Patrick LöberAI Engineer Europe 20262026
Stephen BatifolAI Engineer Europe 20262026
Apoorva JoshiAI Engineer World's Fair 20252025
Romain HuetAI Engineer World's Fair 20242024
Cedric VidalAI Engineer World's Fair 20242024
Karan GoelAI Engineer World's Fair 20242024
Ben HylakAI Engineer World's Fair 20242024
Stefania DrugaAI Engineer World's Fair 20242024
Allen PikeAI Engineer World's Fair 20262026
Abed MatiniAI Engineer World's Fair 20262026
Jeremy Silva, Chris HernandezAI Engineer World's Fair 20252025
Vision: Zero Bugs

Metadata candidate

Johann Schleier-SmithAI Engineer Code 20252025
Shelby HeineckeAI Engineer World's Fair 20242024
Barry Zhang, Mahesh MuragAI Engineer Code 20252025
Ido SalomonAI Engineer Europe 20262026
Dan Fu, Olive SongAI Engineer World's Fair 20262026
AGI: The Path Forward

Metadata candidate

Eiso Kant, Jason WarnerAI Engineer Code 20252025
Rajat ShahAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20252025
swyxAI Engineer World's Fair 20242024
Natalie SerrinoAI Engineer Code 20252025
Ivan BurazinAI Engineer World's Fair 20252025
Grace IsfordAI Engineer Summit 20252025
Rajiv ChandegraAI Engineer World's Fair 20262026
Sunny MadraAI Engineer World's Fair 20242024
Paige BaileyAI Engineer Europe 20262026
Eliza Cabrera, Jeremy SilvaAI Engineer World's Fair 20252025
Cedric Vidal, David Smith, Miguel MartinezAI Engineer World's Fair 20242024
Du’An Lightfoot, Banjo ObayomiAI Engineer World's Fair 20252025
Jerry LiuAI Engineer World's Fair 20252025
Tom RedmanAI Engineer World's Fair 20242024
Marlene Mhangami, Liam HamptonAI Engineer Europe 20262026
Liam McGarrigleAI Engineer Europe 20262026
Rachna SrivastavaAI Engineer World's Fair 20252025
Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Ari MorcosAI Engineer World's Fair 20262026
Defying Gravity

Metadata candidate

Kevin HouAI Engineer Code 20252025
Keegan McCallumAI Engineer World's Fair 20252025
Kevin MaduraAI Engineer Code 20252025
Joseph Wang, SidAI Engineer World's Fair 20262026
Ofer MendelevitchAI Engineer Code 20252025
Benjamin FletcherAI Engineer World's Fair 20242024
Cormac BrickAI Engineer Europe 20262026
Abhishek BhardwajAI Engineer World's Fair 20262026
Daniel Kim, Daria SobolevaAI Engineer World's Fair 20252025
Akram BaharloueiAI Engineer World's Fair 20262026
Thor SchaeffAI Engineer Europe 20262026
Cassidy HardinAI Engineer Europe 20262026
Omar SansevieroAI Engineer Europe 20262026
Ruben CasasAI Engineer Europe 20262026
Keegan McCallumAI Engineer World's Fair 20262026
Giving a Voice to AI Agents

Metadata candidate

Scott StephensonAI Engineer World's Fair 20242024
John PhamAI Engineer World's Fair 20252025
Audry HsuAI Engineer Europe 20262026
Andreas KolleggerAI Engineer World's Fair 20252025
Nik PashAI Engineer Code 20252025
Vasant KearneyAI Engineer World's Fair 20262026
How Deep Research Works

Metadata candidate

Mukund Sridhar, Aarush SelvanAI Engineer Summit 20252025
Raia HadsellAI Engineer Europe 20262026
Joe ReeveAI Engineer Europe 20262026
Patricija ŽemaitytėAI Engineer World's Fair 20262026
Amol KapoorAI Engineer World's Fair 20262026
Lachlan Ainley, Humza IqbalAI Engineer World's Fair 20242024
Yu SuAI Engineer World's Fair 20262026
Philip Kiely, Yineng ZhangAI Engineer World's Fair 20252025
Alex LissAI Engineer World's Fair 20252025
Judging LLMs

Metadata candidate

Alex VolkovAI Engineer World's Fair 20242024
Kent C. DoddsAI Engineer World's Fair 20252025
Rukma SenAI Engineer World's Fair 20242024
Ben HolmesAI Engineer World's Fair 20262026
Shafik Quoraishee, Joanne SongAI Engineer World's Fair 20262026
Lin Qiao, Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Drasko ProfirovicAI Engineer World's Fair 20262026
Minimax M2

Metadata candidate

Olive SongAI Engineer Code 20252025
Move Fast Break Nothing

Metadata candidate

Dedy KredoAI Engineer Summit 20232023
Rémi LoufAI Engineer World's Fair 20242024
Simon WillisonAI Engineer World's Fair 20242024
Ofer MendelevitchAI Engineer World's Fair 20252025
Phil NashAI Engineer Europe 20262026
Antje BarthAI Engineer World's Fair 20262026
Diego RodriguezAI Engineer World's Fair 20252025
Benjamin SteinAI Engineer World's Fair 20242024
Steven MoonAI Engineer Summit 20252025
Jason LiuAI Engineer World's Fair 20242024
Andres MarafiotiAI Engineer Europe 20262026
Stefania DrugaAI Engineer World's Fair 20252025
Chad Bailey, Brian JohnsonAI Engineer World's Fair 20252025
Eugene YanAI Engineer World's Fair 20252025
Will BrownAI Engineer World's Fair 20262026
Anton TroynikovAI Engineer Summit 20232023
Robotics: why now?

Metadata candidate

Quan Vuong, Jost Tobias SpringenbergAI Engineer World's Fair 20252025
Adrien GrondinAI Engineer Europe 20262026
Fouad MatinAI Engineer World's Fair 20252025
Onur SolmazAI Engineer Europe 20262026
Shawn JanseparAI Engineer World's Fair 20242024
Calvin Qi, Chang SheAI Engineer World's Fair 20252025
Adrian BertagnoliAI Engineer Europe 20262026
See, Hear, Speak, Draw

Metadata candidate

Logan Kilpatrick, Simón FishmanAI Engineer Summit 20232023
Merve NoyanAI Engineer Europe 20262026
Arjun Desai, Rohit TalluriAI Engineer World's Fair 20252025
Kenneth AuchenbergAI Engineer World's Fair 20252025
Annabell SchäferAI Engineer World's Fair 20262026
Cedric ClyburnAI Engineer World's Fair 20262026
Rob CheungAI Engineer World's Fair 20242024
Devansh TandonAI Engineer World's Fair 20252025
Barr YaronAI Engineer World's Fair 20252025
Michele CatastaAI Engineer Code 20252025
Dani Grant, Chelcie TaylorAI Engineer World's Fair 20252025
Travis FrisingerAI Engineer World's Fair 20252025
Junyang LinAI Engineer World's Fair 20252025
Linus LeeAI Engineer Summit 20232023
Chang She, Noah ShpakAI Engineer World's Fair 20242024
Jesse HanAI Engineer World's Fair 20252025
Hassan El MghariAI Engineer World's Fair 20262026
Kwindla Kramer, Kwindla Hultman KramerAI Engineer World's Fair 20262026
Arturo NunezAI Engineer World's Fair 20262026
Filip MakraduliAI Engineer World's Fair 20252025
Frank LiuAI Engineer World's Fair 20252025
Alberto RomeroAI Engineer Code 20252025
Paul Klein IVAI Engineer World's Fair 20252025
Thinking Deeper in Gemini

Metadata candidate

Jack RaeAI Engineer World's Fair 20252025
MuhtesemAI Engineer Summit 20252025
tldraw computer

Metadata candidate

Steve RuizAI Engineer World's Fair 20252025
Sangwu LeeAI Engineer World's Fair 20262026
Jeff SchomayAI Engineer Summit 20232023
James LeAI Engineer World's Fair 20262026
Sidney PrimasAI Engineer World's Fair 20262026
Jyh-Jing HwangAI Engineer World's Fair 20252025
Tobin SouthAI Engineer World's Fair 20252025
Annika Brundyn, Aastha JhunjhunwalaAI Engineer World's Fair 20252025
Mukuntha Narayanan, Han WangAI Engineer World's Fair 20252025
Tom Shapland, PhDAI Engineer World's Fair 20252025
Dr. Jasper ZhangAI Engineer World's Fair 20252025
Karina NguyenAI Engineer Summit 20232023
Balázs HorváthAI Engineer World's Fair 20262026
Ziv IlanAI Engineer Europe 20262026
Veronica HylakAI Engineer World's Fair 20262026
Jingxiang "JX" MoAI Engineer World's Fair 20252025
Bertrand CharpentierAI Engineer Europe 20262026
Bala RamdossAI Engineer World's Fair 20262026
AI Engineering 101

Metadata candidate

Noah HeinAI Engineer Summit 20232023
Siddharth AhujaAI Engineer World's Fair 20252025
Ben KusAI Engineer World's Fair 20252025
Jamie Neuwirth, Zack WittenAI Engineer World's Fair 20242024
Andrew ThompsonAI Engineer World's Fair 20252025
Ben HolmesAI Engineer World's Fair 20252025
Jedrick Kosinski, ComfyAnonymousAI Engineer World's Fair 20252025
Dhruv BatraAI Engineer World's Fair 20262026
Content Is Code

Metadata candidate

Matt PalmerAI Engineer World's Fair 20262026
Guillaume VernadeAI Engineer Europe 20262026
Ido Salomon, Liad YosefAI Engineer World's Fair 20262026
Jon PeckAI Engineer World's Fair 20252025
Vaidas RazgaitisAI Engineer World's Fair 20262026
Stephan SteinfurtAI Engineer Europe 20262026
Neil Dwyer, Jack DwyerAI Engineer World's Fair 20252025
Nuno CamposAI Engineer Europe 20262026
Zack Reneau-WedeenAI Engineer Summit 20252025
Corey GallonAI Engineer World's Fair 20262026
The Weekend AI Engineer

Metadata candidate

Hassan El MghariAI Engineer Summit 20232023
Kathleen KenealyAI Engineer World's Fair 20242024
Hassan El MghariAI Engineer World's Fair 20252025
Mark BissellAI Engineer World's Fair 20252025
Olive SongEvent unknownUnknown
Jordi SprangerAI Engineer NYC 2026 Online Talks2026
Armen AghajanyanAI Engineer World's Fair 20262026
Krishna Prasad SrinivasanAI Engineer World's Fair 20262026
Cody MenefeeAI Engineer World's Fair 20262026
Andrew DaiAI Engineer World's Fair 20262026
Rafael LeviAI Engineer World's Fair 20262026
Thor SchaeffAI Engineer World's Fair 20262026
Merve NoyanAI Engineer World's Fair 20262026
Suchet BargotiAI Engineer World's Fair 20262026
Aditya GautamAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
23 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
173 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography

    https://graphics.stanford.edu/courses/cs233-21-spring/ReferencedPapers/ransac.pdf

  2. Foundations of Computer Vision: Stereo Vision

    https://visionbook.mit.edu/3d_scene_understanding_stereo.html

  3. Torchvision Bounding-Box Geometry Kernels

    https://raw.githubusercontent.com/pytorch/vision/main/torchvision/transforms/v2/functional/_geometry.py

  4. Foundations of Computer Vision: Optical Flow Estimation

    https://visionbook.mit.edu/optical_flow.html

  5. MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection

    https://www.mvtec.com/fileadmin/Redaktion/mvtec.com/05_research_teaching/datasets/mvtec_ad.pdf

  6. CS231n: Object Detection and Image Segmentation

    https://cs231n.stanford.edu/slides/2024/lecture_9.pdf

  7. MAttNet: Modular Attention Network for Referring Expression Comprehension

    https://arxiv.org/html/1801.08186

  8. Simple Online and Realtime Tracking

    https://arxiv.org/html/1602.00763

  9. Dense-Captioning Events in Videos

    https://arxiv.org/html/1705.00754

  10. Fast, efficient, reliable: Artificial intelligence in BMW Group Production

    https://www.press.bmwgroup.com/global/article/detail/T0298650EN/fast-efficient-reliable:-artificial-intelligence-in-bmw-group-production?showMedia=video

  11. OpenCV: TextDetectionModel and TextRecognitionModel

    https://docs.opencv.org/4.13.0/d4/d43/tutorial_dnn_text_spotting.html

  12. Foundations of Computer Vision: Imaging

    https://visionbook.mit.edu/imaging.html

  13. Physically Based Rendering: Film and Imaging

    https://www.pbr-book.org/4ed/Cameras_and_Film/Film_and_Imaging

  14. OpenCV: Image File Reading and Writing

    https://docs.opencv.org/4.12.0/d4/da8/group__imgcodecs.html

  15. Foundations of Computer Vision: Image Sampling and Aliasing

    https://visionbook.mit.edu/sampling_and_aliasing.html

  16. Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection

    https://arxiv.org/html/2202.06934

  17. Machine Perception of Three-Dimensional Solids

    https://s3.amazonaws.com/images.designworldonline.com/CADhistory/Lawrence%20Roberts%2033959125.pdf

  18. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information

    https://fliphtml5.com/yildh/cecq/Marr%5D_Vision_A_Computational_Investigation/

  19. Determining Optical Flow

    https://www.sci.utah.edu/~gerig/CS6320-S2012/Materials/Horn-Schunck-1981-orig-article.pdf

  20. Handwritten Digit Recognition with a Back-Propagation Network

    https://proceedings.neurips.cc/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf

  21. Gradient-Based Learning Applied to Document Recognition

    https://gwern.net/doc/ai/nn/cnn/1998-lecun.pdf

  22. Distinctive Image Features from Scale-Invariant Keypoints

    https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf

  23. The PASCAL Visual Object Classes (VOC) Challenge

    https://www.robots.ox.ac.uk/~vgg/publications/2010/Everingham10/everingham10.pdf

  24. ImageNet: A Large-Scale Hierarchical Image Database

    https://image-net.org/static_files/papers/imagenet_cvpr09.pdf

  25. ImageNet Classification with Deep Convolutional Neural Networks

    https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf

  26. Two-Stream Convolutional Networks for Action Recognition in Videos

    https://arxiv.org/pdf/1406.2199

  27. Fully Convolutional Networks for Semantic Segmentation

    https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf

  28. Deep Residual Learning for Image Recognition

    https://arxiv.org/html/1512.03385

  29. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    https://arxiv.org/html/2010.11929

  30. Learning Transferable Visual Models From Natural Language Supervision

    https://arxiv.org/html/2103.00020

  31. Segment Anything

    https://arxiv.org/html/2304.02643

  32. An Iterative Image Registration Technique with an Application to Stereo Vision

    https://idl.uw.edu/living-papers-paper/lucas-kanade/

  33. CS231n: Convolutional Networks

    https://cs231n.github.io/convolutional-networks/

  34. PyTorch: Training a Classifier

    https://docs.pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

  35. Foundations of Computer Vision: Convolutional Neural Nets

    https://visionbook.mit.edu/convolutional_neural_nets.html

  36. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

    https://arxiv.org/html/2103.14030

  37. How Transformers Finally Ate Vision

    https://www.youtube.com/watch?v=VhfAVA3BG2I&t=415

  38. How Transformers Finally Ate Vision

    https://www.youtube.com/watch?v=VhfAVA3BG2I&t=523

  39. PyTorch: BCEWithLogitsLoss

    https://docs.pytorch.org/docs/2.13/generated/torch.nn.BCEWithLogitsLoss.html

  40. 120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson

    https://www.youtube.com/watch?v=OimPoLxioYg&t=55

  41. 120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson

    https://www.youtube.com/watch?v=OimPoLxioYg&t=805

  42. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=212

  43. Towards Open Set Deep Networks

    https://arxiv.org/html/1511.06233

  44. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

    https://arxiv.org/html/1506.01497

  45. You Only Look Once: Unified, Real-Time Object Detection

    https://arxiv.org/html/1506.02640

  46. Feature Pyramid Networks for Object Detection

    https://arxiv.org/html/1612.03144

  47. Torchvision Non-Maximum Suppression

    https://docs.pytorch.org/vision/stable/generated/torchvision.ops.nms.html

  48. End-to-End Object Detection with Transformers

    https://arxiv.org/html/2005.12872

  49. COCO Official Detection Evaluator

    https://raw.githubusercontent.com/cocodataset/cocoapi/master/PythonAPI/pycocotools/cocoeval.py

  50. Panoptic Segmentation

    https://arxiv.org/pdf/1801.00868

  51. U-Net: Convolutional Networks for Biomedical Image Segmentation

    https://arxiv.org/html/1505.04597

  52. Mask R-CNN

    https://arxiv.org/html/1703.06870

  53. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

    https://www.youtube.com/watch?v=C13jiFWNuo8&t=287

  54. SAM 3: Segment Anything with Concepts

    https://arxiv.org/html/2511.16719

  55. Detectron2: Use Custom Datasets

    https://detectron2.readthedocs.io/en/latest/tutorials/datasets.html

  56. Foundations of Computer Vision: Camera Modeling and Calibration

    https://visionbook.mit.edu/imaging_geometry.html

  57. A Flexible New Technique for Camera Calibration

    https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr98-71.pdf

  58. OpenCV: Epipolar Geometry

    https://docs.opencv.org/4.13.0/da/de9/tutorial_py_epipolar_geometry.html

  59. Depth Anything V2

    https://arxiv.org/html/2406.09414

  60. Foundations of Computer Vision: Multiview Geometry and Structure from Motion

    https://visionbook.mit.edu/multiview.html

  61. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

    https://arxiv.org/html/2303.05499

  62. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

    https://arxiv.org/html/1612.06890

  63. GRES: Generalized Referring Expression Segmentation

    https://arxiv.org/html/2306.00968

  64. Visual Instruction Tuning

    https://arxiv.org/html/2304.08485

  65. Moondream: how does a tiny vision model slap so hard?

    https://www.youtube.com/watch?v=T7sxvrJLJ14&t=217

  66. Moondream: how does a tiny vision model slap so hard?

    https://www.youtube.com/watch?v=T7sxvrJLJ14&t=384

  67. A Simple Framework for Contrastive Learning of Visual Representations

    https://proceedings.mlr.press/v119/chen20j/chen20j.pdf

  68. Foundations of Computer Vision: Training for Robustness and Generality

    https://visionbook.mit.edu/data_augmentation.html

  69. Masked Autoencoders Are Scalable Vision Learners

    https://arxiv.org/html/2111.06377

  70. Emerging Properties in Self-Supervised Vision Transformers

    https://arxiv.org/html/2104.14294

  71. DINOv2: Learning Robust Visual Features without Supervision

    https://arxiv.org/html/2304.07193

  72. DINOv3

    https://arxiv.org/html/2508.10104

  73. Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution

    https://arxiv.org/html/2202.10054

  74. Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models

    https://arxiv.org/html/2505.20612v1

  75. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=990

  76. Semantic Amodal Segmentation

    https://arxiv.org/pdf/1509.01329

  77. LVIS: A Dataset for Large Vocabulary Instance Segmentation

    https://www.lvisdataset.org/assets/lvis_v0.5.pdf

  78. Google ML Crash Course: Class-Imbalanced Datasets

    https://developers.google.com/machine-learning/crash-course/overfitting/imbalanced-datasets

  79. Torchvision Segmentation Reference Transforms

    https://github.com/pytorch/vision/blob/main/references/segmentation/transforms.py

  80. COCO-WholeBody: Annotation File Format

    https://github.com/jin-s13/COCO-WholeBody/blob/master/data_format.md

  81. Detectron2: Keypoint Annotation Transformations

    https://raw.githubusercontent.com/facebookresearch/detectron2/main/detectron2/data/detection_utils.py

  82. FFmpeg: AVFrame Struct Reference

    https://www.ffmpeg.org/doxygen/trunk/structAVFrame.html

  83. Retro-Actions: Learning ‘Close’ by Time-Reversing ‘Open’ Videos

    https://arxiv.org/html/1909.09422

  84. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

    https://arxiv.org/html/1705.07750

  85. Is Space-Time Attention All You Need for Video Understanding?

    https://arxiv.org/html/2102.05095

  86. HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

    https://arxiv.org/html/2009.07736

  87. Simple Online and Realtime Tracking with a Deep Association Metric

    https://arxiv.org/html/1703.07402

  88. SAM 2: Segment Anything in Images and Videos

    https://arxiv.org/html/2408.00714

  89. Online Action Detection

    https://arxiv.org/html/1604.06506

  90. Ego4D: Around the World in 3,000 Hours of Egocentric Video

    https://www.cs.utexas.edu/~grauman/papers/ego4d-cvpr2022.pdf

  91. SlowFast Networks for Video Recognition

    https://arxiv.org/html/1812.03982

  92. Online Detection of Action Start in Untrimmed, Streaming Videos

    https://merl.com/publications/docs/TR2018-109.pdf

  93. VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

    https://aclanthology.org/2021.emnlp-main.544.pdf

  94. TempCompass: Do Video LLMs Really Understand Videos?

    https://arxiv.org/html/2403.00476

  95. LLaVA-OneVision: Easy Visual Task Transfer

    https://arxiv.org/html/2408.03326

  96. Selective Classification for Deep Neural Networks

    https://arxiv.org/html/1705.08500

  97. WILDS: A Benchmark of in-the-Wild Distribution Shifts

    https://arxiv.org/html/2012.07421

  98. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

    https://arxiv.org/html/1811.12231

  99. ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression

    https://arxiv.org/html/2509.20234

  100. Noise or Signal: The Role of Image Backgrounds in Object Recognition

    https://arxiv.org/html/2006.09994

  101. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

    https://arxiv.org/html/1903.12261

  102. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

    https://arxiv.org/html/1906.02530

  103. Cityscapes Official Pixel-Level Evaluator

    https://raw.githubusercontent.com/mcordts/cityscapesScripts/master/cityscapesscripts/evaluation/evalPixelLevelSemanticLabeling.py

  104. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

    https://arxiv.org/html/1606.04797

  105. Boundary IoU: Improving Object-Centric Image Segmentation Evaluation

    https://arxiv.org/html/2103.16562

  106. COCO Official Evaluator: computeOks

    https://raw.githubusercontent.com/cocodataset/cocoapi/master/PythonAPI/pycocotools/cocoeval.py

  107. The Fourth Monocular Depth Estimation Challenge

    https://arxiv.org/html/2504.17787

  108. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

    https://arxiv.org/html/2204.03162

  109. Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking

    https://arxiv.org/html/1609.01775

  110. ActivityNet Official Temporal Detection Evaluator

    https://raw.githubusercontent.com/activitynet/ActivityNet/master/Evaluation/eval_detection.py

  111. ActEV Sequestered Data Leaderboard Evaluation Plan

    https://actev.nist.gov/pub/ActEV_SDL_EvaluationPlan_081219.pdf

  112. ActEV 2021 Sequestered Data Leaderboard Evaluation Plan

    https://www.readkong.com/page/actev-2021-sequestered-data-leaderboard-sdl-evaluation-2639984

  113. scikit-learn: Cross-Validation for Grouped and Time-Dependent Data

    https://scikit-learn.org/stable/modules/cross_validation.html

  114. WILDS Release Notes: iWildCam v1.0 to v2.0

    https://github.com/p-lambda/wilds/releases

  115. Google ML Guides: Data Quality and Interpretation

    https://developers.google.com/machine-learning/guides/data-traps/quality?hl=en

  116. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

    https://www.youtube.com/watch?v=C13jiFWNuo8&t=379

  117. A framework for flexible and reconfigurable vision inspection systems

    https://link.springer.com/article/10.1007/s00170-023-12175-6

  118. From Lab to Factory: Pitfalls and Guidelines for Self-/Unsupervised Defect Detection on Low-Quality Industrial Images

    https://arxiv.org/html/2506.16890

  119. Dynamic World, Near real-time global 10 m land use land cover mapping

    https://www.nature.com/articles/s41597-022-01307-4.pdf

  120. Good practices for estimating area and assessing accuracy of land change

    https://samv.elearning.unipd.it/pluginfile.php/175898/mod_resource/content/0/articolo_oloffson.pdf

  121. Moondream: how does a tiny vision model slap so hard?

    https://www.youtube.com/watch?v=T7sxvrJLJ14&t=897

  122. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=70

  123. Google ML Crash Course: Overfitting

    https://developers.google.com/machine-learning/crash-course/overfitting/overfitting

  124. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=358

  125. Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters

    https://arxiv.org/html/2411.03312

  126. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=929

  127. Moondream: how does a tiny vision model slap so hard?

    https://www.youtube.com/watch?v=T7sxvrJLJ14&t=463

  128. State of the Union: Why Local, Why Now

    https://www.youtube.com/watch?v=KB41dTlX1Uc&t=2165

  129. How Transformers Finally Ate Vision

    https://www.youtube.com/watch?v=VhfAVA3BG2I&t=121

  130. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=282

  131. Building Closed-Loop Evals for a Multimodal Agent at Uber Scale

    https://www.youtube.com/watch?v=31GUkCBD-Uc&t=668