Contents
  1. What learning changes
    1. Behavior fitted from examples
      1. Determinism is a separate choice
    2. Examples, features, and targets
      1. Choose the target before the algorithm
      2. Labels are measurements
    3. Parameters and inductive assumptions
      1. Finite data leave choices open
      2. How the developments connect
  2. From signal to fitted model
    1. Sources of supervision
      1. Targets constructed from observations
    2. Loss and the objective
      1. The penalty changes influence
      2. Optimization can succeed at the wrong task
    3. Parameter updates
      1. Backpropagation computes; the optimizer updates
      2. Fitting is not inference
  3. Evidence beyond the fitted sample
    1. Generalization to new cases
      1. What each score can support
      2. Rows are not always independent evidence
    2. Validation without leakage
      1. Selection can overfit too
      2. Match the boundary to the claim
      3. Baselines and uncertainty
    3. Underfitting, overfitting, and regularization
      1. Constrain the fit, not the evidence
  4. Uncertainty, decisions, and change
    1. Sources of predictive uncertainty
      1. Variation versus limited knowledge
      2. Different intervals answer different questions
    2. From probabilities to actions
      1. The same probability can justify different actions
      2. Rare events produce surprising alert streams
    3. When deployment conditions shift
      1. Different changes require different evidence
      2. The model can change what it sees
  5. Prediction is not intervention
    1. Association, action, and causal effect
      1. A common cause can reverse the lesson
      2. What intervention evidence requires
      3. Prediction creates feedback
      4. Four claims, four evidence burdens
  6. Check understanding
  7. Open questions
  8. Selected talks
  9. References
  10. Talk library
← All topics

ML fundamentals

Machine learning is useful when the behavior you need is easier to demonstrate with examples than to specify as a complete set of rules. A model remains ordinary computation: it receives inputs and produces outputs. What changes is the origin of some of its behavior. Instead of an engineer writing every decision boundary, training selects parameter values that reduce a chosen objective on recorded examples. This exchange creates new engineering obligations. You must define what is predicted, prevent unavailable information from entering the inputs, choose what errors training rewards, and test the resulting behavior on cases that did not influence it. Even then, a model’s probability needs interpretation, deployment conditions can change, and a predictive association does not establish that manipulating an input will improve the outcome.

Part I — What learning changes

Behavior fitted from examples

An explicitly programmed rule gets its decision behavior from authored control flow. A learned predictor gets part of its behavior from parameters fitted to examples. For a request-duration predictor, an engineer might choose the inputs and model family, while fitting determines coefficients that connect request properties to a duration estimate. The resulting model is still a function that software can call.

QuestionAuthored ruleLearned predictor
Where does behavior come from?Conditions written by a programmerParameters fitted from examples
What varies per request?Input valuesInput values
What changes during training?Nothing unless code is editedLearned parameters
What happens during ordinary inference?The rule executesThe fitted function executes
What establishes correctness?Contract and behavioral testsContract checks plus statistical evidence on relevant new cases

Determinism is a separate choice

Learning does not make execution inherently random. A fitted model can return the same output for the same input. Separate three choices: randomness used while fitting, optional sampling used to select a generated output, and nondeterministic numerical implementations. Seeding and deterministic-algorithm settings address different parts of that problem and may still vary across software versions or hardware.

A valid interface is also different from a valid prediction. JSON schema validation can establish that predicted_seconds is a number; it cannot establish that the number is close to the eventual duration. Software Engineering Fundamentals owns the fuller treatment of explicit contracts. Here the important point is that learned behavior adds an empirical correctness question to the usual software questions.

Examples, features, and targets

A learning problem begins by fixing the unit of prediction. One example is one unit about which the model should learn: a service request, an image, a document, or a customer session. Its features are observations available when the prediction must be made. Its label, or target, is the recorded answer training asks the model to predict. A collection used for fitting is the training set.

Choose the target before the algorithm

Predicting a numerical target, such as duration in seconds, is regression. Predicting one category from a set, such as completed, canceled, or timed_out, is classification. Encoding those categories as 0, 1, and 2 does not turn the task into regression; the numbers are identifiers unless their distances have a defined meaning.

Consider an invented request record. At submission time, the system knows the operation type, payload size, service version, and queue depth. Completion duration appears later. Duration can be the target, but using it as an input to predict itself would be leakage. Log-derived setup or execution durations pose the same timing problem: a field can be useful for diagnosis after completion yet invalid as a prediction-time feature.

A target appears after the prediction point

Example

The same request persists, but completion information is introduced only after the model had to predict.

1 / 3 · At submission

The request identity persists; only submission-time fields are available as possible prediction inputs.

New this stepRequest R-17The stable request entity.
New this stepSubmission fieldsOperation, payload size, service version, and queue depth recorded at prediction time.
In this invented record, the same request acquires completion information only later. The prediction uses submission-time fields; the later observed duration is its target, never a prediction input.
Read the diagram as text
  • Request R-17. The stable request entity.
  • Submission fields. Operation, payload size, service version, and queue depth recorded at prediction time.
  • Duration prediction. A representation derived from the submission fields, not a property already stored on the request.
  • Completion state. A later state of the same request.
  • Observed duration. The target becomes available only after completion.
  • Request R-17 → Submission fields: has recorded state.
  • Submission fields → Duration prediction: data input.
  • Request R-17 → Completion state: later state.
  • Completion state → Observed duration: records target.
  1. At submission. The request identity persists; only submission-time fields are available as possible prediction inputs. Active: Request R-17, Submission fields. New: Request R-17, Submission fields.
  2. Make the prediction. The model derives a duration estimate from the available fields. Active: Request R-17, Submission fields, Duration prediction. New: Duration prediction.
  3. Observe the outcome. Completion adds the target while preserving the original request identity and fields. Active: Request R-17, Submission fields, Duration prediction, Completion state, Observed duration. New: Completion state, Observed duration.
FieldAvailable at submission?Role in duration prediction
request_idYesIdentity; usually not a predictive feature
payload_bytesYesPossible feature
service_versionYesPossible feature
queue_depthYesPossible feature if recorded at the prediction time
completion_durationNoTarget
post-run error traceNoLater diagnostic evidence, not a submission-time feature

Labels are measurements

A recorded target is not automatically the desired truth. It may arrive late, contain measurement error, encode an ambiguous policy choice, or exist only for cases selected by an earlier decision. If a dataset contains only completed requests, it may omit canceled and unfinished requests that future traffic includes. Likewise, an exact label match can be deterministic while the label itself remains disputable or depends on information withheld from the model. Collection, provenance, and labeling practice are developed in Data Quality and Curation.

Parameters and inductive assumptions

Suppose a duration predictor uses payload size xx:

y^=b+wx\hat{y}=b+wx

Here xx changes with each request. The coefficient ww and intercept bb are learned parameters that remain fixed during ordinary prediction. An engineer may instead choose a regularization strength, tree depth, number of layers, or learning rate. Such a setting is a hyperparameter: it governs the candidate functions or the fitting procedure rather than being updated by the ordinary parameter-fitting loop.

Finite data leave choices open

A model family is the set of functions fitting may choose from. Linear regression can choose different lines. A decision tree can choose different feature thresholds and leaf values. A neural network composes parameterized transformations and may learn intermediate features rather than requiring every useful representation to be authored. Embeddings and Representation Learning develops that last idea. Two functions can fit the same observations yet disagree elsewhere. An inductive bias is the preference that makes one continuation easier to select than another.

Historical foundations of fitted behavior

  1. 1805Legendre publishes least squaresSelects unknown parameters by minimizing the sum of squared observational errors.Sources & context

    Contributors: Adrien-Marie Legendre

    What changed: Made parameter selection from imperfect, overdetermined observations explicit. This is a publication milestone, not a sole-invention claim.

  2. 1958Rosenblatt’s perceptronStudies recognition and generalization in a network whose effective connections change through experience.Sources & context

    Contributors: Frank Rosenblatt, Cornell Aeronautical Laboratory

    What changed: Made experience-dependent classification concrete through reinforcement dynamics that altered later responses.

  3. 1986Rumelhart, Hinton, and Williams on back-propagated errorsUses output errors to train intermediate representations in multilayer networks.Sources & context

    Contributors: David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams

    What changed: Showed how the chain rule could efficiently provide weight sensitivities so hidden units could learn useful features from desired outputs.

Notice the progression from selecting parameters under observational error, to adapting classification connections through experience, to learning intermediate representations from output errors. Spacing is not to scale.

Observed points do not determine one continuation

Example

Two functions fit every observed point yet disagree at an unobserved input.

Two sample-consistent functions

Both candidates agree at x=1, 2, and 4 but differ at the unobserved query x=3.

0123402468Input x (dimensionless)Predicted target (dimensionless)Linear candidateCurved candidateUnobserved query x=3Observed examplesx=3: unobserved64.8
  • 1. Linear candidate
  • 2. Curved candidate
  • 3. Unobserved query x=3
  • 4. Observed examples
Read coordinates and regions as data

X: 0–4 dimensionless; Y: 0–8 dimensionless, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Linear candidate (polyline)

(0, 0); (1, 2); (2, 4); (3, 6); (4, 8)

Curved candidate (polyline)

(0, 1); (1, 2); (2, 4); (3, 4.8); (4, 8)

Unobserved query x=3 (polyline)

(3, 0); (3, 6.6)

Observed examples (points)

(1, 2); (2, 4); (4, 8)

x=3: unobserved: (3, 7.25)

6: (3.15, 6.2)

4.8: (3.15, 4.6)

Both candidates fit the three invented observations exactly. At the unobserved x=3, the line predicts 6 and the alternative predicts 4.8. A model family or regularizer favors a continuation; neither is established as correct by these observations.

Architecture, feature representation, regularization, and optimization all contribute to inductive bias. Finite observations do not uniquely determine behavior everywhere; the engineering question is whether the preference fits the intended setting.

The same observations can support different fitted representations. Increasing the range of candidate functions gives fitting more choices, but still leaves the engineer responsible for the objective and the conditions under which predictions will be judged.

How the developments connect

These approaches coexist with earlier statistical methods. Richer representations extend what can be fitted; they do not remove observational error or make the resulting predictor correct outside its assessed setting.

Part II — From signal to fitted model

Sources of supervision

A learning signal tells a fitting procedure how current behavior should change. In supervised learning, each example carries a supplied target: request fields paired with an observed duration, or an image paired with a class. The target need not be perfect, but it is explicitly treated as the answer for that training example.

Targets constructed from observations

In self-supervised learning, the data provide their own constructed targets. BERT, for example, corrupts selected positions in ordinary text and trains the model to recover the original content. The visible text becomes the input and the preserved original values become targets. No person must label every position, but the designer still chooses what to hide, what context is visible, and why recovering it should create reusable behavior.

This is distinct from using a large teacher model to generate labels. Teacher-generated labels are synthetic supervision and may reproduce teacher failures; they are not self-supervised merely because humans did not write them individually. Zeta2’s reported pipeline, for example, checked teacher edit predictions for task-specific violations and attempted repairs before using them as student targets.

Other learning signals encode different structures. Clustering can optimize compactness without supplied category labels. Reinforcement learning uses rewards from interaction rather than demonstrated actions. These signals are not interchangeable: each defines what information rewards a parameter change. Specialized corpus objectives belong in Pretraining and Midtraining; demonstrations, preferences, and rewards are developed in Post-training and Alignment.

Loss and the objective

A loss assigns a numerical penalty to a prediction relative to its target. A training objective aggregates losses across examples and may add weights, constraints, or regularization. The fitting procedure does not understand the prose description of your goal; it receives pressure from this computable quantity.

MAE⁡=1n∑i=1n∣yi−y^i∣,MSE⁡=1n∑i=1n(yi−y^i)2\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|,\qquad \operatorname{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2

The penalty changes influence

Both objectives prefer small errors, but they apply different pressure. With absolute error, doubling an error doubles its contribution. With squared error, doubling an error quadruples its contribution. A single large duration miss therefore influences mean squared error much more strongly. That may be desirable when large misses dominate an operational budget, or undesirable when rare measurement failures should not control the fit. The objective encodes this choice; its name does not settle it.

How one residual changes the objective

For one duration prediction, the residual is observed duration minus predicted duration, in seconds. A negative residual means the prediction was too high.

Use the arrow keys to change the residual by half a second.

Residual 2 seconds: absolute penalty 2 seconds; squared penalty 4 seconds squared.

One example’s penalty contributions
ResidualAbsolute penalty (s)Squared penalty (s²)
Current: 2 s24
Doubled: 4 s416

Doubling the residual magnitude multiplies the absolute penalty by 2 and the squared penalty by 4. Reversing its sign changes neither penalty.

These are terms for one example, before summing and dividing by the number of examples. MAE has duration units; MSE has squared-duration units. Their raw numeric sizes do not say which objective is better.

Compare a residual with twice its magnitude, then reverse its sign. These are per-example penalty contributions, not the full dataset means.
L=−1n∑i=1n[yilog⁡pi+(1−yi)log⁡(1−pi)]L=-\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log p_i+(1-y_i)\log(1-p_i)\right]

For binary classification, log loss rewards probability assigned to the recorded class and heavily penalizes confident mistakes. Weighting examples changes which cases dominate the aggregate. Adding λ∑jwj2\lambda\sum_j w_j^2 favors smaller coefficients. These choices affect the selected predictor even when the examples are unchanged.

Optimization can succeed at the wrong task

Training loss remains a proxy. A model can reduce teacher disagreement without becoming more useful, optimize click labels without improving user welfare, or earn a learned preference score without becoming factually correct. Application metrics and decision consequences must therefore be evaluated separately from the quantity optimized during fitting.

Parameter updates

Gradient-based fitting repeats four operations: make predictions, compute the objective, calculate how locally sensitive that objective is to each parameter, and update the parameters. A gradient collects those local sensitivities. It does not identify a symbolic business rule to edit; it says how a small numerical parameter change would affect the current objective.

θt+1=θt−η 1m∑i=1m∇θℓi(θt)\theta_{t+1}=\theta_t-\eta\,\frac{1}{m}\sum_{i=1}^{m}\nabla_\theta \ell_i(\theta_t)

A minibatch is the subset of training examples used for one parameter update; in the equation, mm is its size. The parameter vector is θ\theta, and η\eta is the learning rate controlling update scale. Small minibatches provide noisier gradient estimates but can make frequent updates; larger ones can use vectorized hardware efficiently. Changing batch size therefore changes both optimization behavior and execution.

Backpropagation computes; the optimizer updates

For composed differentiable operations, backpropagation applies the chain rule efficiently to compute parameter gradients. The optimizer then decides how to use those gradients. Plain stochastic gradient descent subtracts a scaled gradient; Adam and other optimizers transform or accumulate gradient information differently. Backpropagation is therefore not the optimizer and does not itself choose the learning rate.

Training adds a return path that inference omits

Backpropagation computes gradients; the optimizer uses them to create the next parameter state.

FeaturesInputs available for the current examples.
Parameters θₜThe parameter values used by the current forward pass.
Forward modelThe authored parameterized computation.
PredictionsOrdinary inference stops here. Training continues to the loss comparison.
TargetsRecorded answers supplied during supervised training.
Batch lossThe objective evaluated from predictions and targets.
BackpropagationComputes the gradient of the loss with respect to θₜ.
OptimizerTraining only: combines θₜ and gradients with its update rule and learning rate to produce θₜ₊₁.
Parameters θₜ₊₁The distinct parameter state produced for the next training iteration.
Features and fitted parameters produce predictions in both modes. Training additionally compares predictions with targets, computes gradients from the loss, and applies an optimizer update. Ordinary inference stops at the prediction.
Read the diagram as text
  • Features. Inputs available for the current examples.
  • Parameters θₜ. The parameter values used by the current forward pass.
  • Forward model. The authored parameterized computation.
  • Predictions. Ordinary inference stops here. Training continues to the loss comparison.
  • Targets. Recorded answers supplied during supervised training.
  • Batch loss. The objective evaluated from predictions and targets.
  • Backpropagation. Computes the gradient of the loss with respect to θₜ.
  • Optimizer. Training only: combines θₜ and gradients with its update rule and learning rate to produce θₜ₊₁.
  • Parameters θₜ₊₁. The distinct parameter state produced for the next training iteration.
  • Features → Forward model: input.
  • Parameters θₜ → Forward model: current values.
  • Forward model → Predictions: forward computation.
  • Predictions → Batch loss: training only: predicted values.
  • Targets → Batch loss: recorded answers.
  • Batch loss → Backpropagation: differentiate.
  • Backpropagation → Optimizer: gradient.
  • Parameters θₜ → Optimizer: previous state.
  • Optimizer → Parameters θₜ₊₁: update.

Illustrative pseudocode

Python-like pseudocode
for batch in training_data:
    predictions = model(batch.features, parameters)
    loss = objective(predictions, batch.targets)
    gradients = backward(loss, parameters)
    parameters = optimizer_step(parameters, gradients)

# Ordinary inference performs only the forward model call.

A large learning rate can cross a low-loss region and land on a worse point; a smaller rate can move toward it more cautiously. But lower training loss is only evidence about the fitted examples. Generalization remains a separate question. Also, not every learner uses gradients: decision trees greedily choose discrete feature splits according to an impurity or loss criterion.

Fitting is not inference

Robbins and Monro’s 1951 stochastic-approximation method established a recursive way to update an estimate from noisy observations under stated conditions. It is a useful historical foundation for noisy iterative updates, but it did not itself describe modern minibatches, neural networks, or backpropagation. During inference, fitted parameters normally remain unchanged. Examples placed in a prompt can still alter a response through the current input; that is the distinct mechanism developed in Prompting and In-Context Learning.

Part III — Evidence beyond the fitted sample

Generalization to new cases

Generalization is useful performance on new cases from the intended setting. Training error averages loss over observed examples. Population error is the expected loss over the underlying distribution of cases the system will encounter. The former is directly computable from the fitted sample; the latter is the quantity deployment usually cares about.

R^(f)=1n∑i=1nℓ(f(xi),yi),R(f)=E(X,Y)∼P[ℓ(f(X),Y)]\widehat{R}(f)=\frac{1}{n}\sum_{i=1}^{n}\ell(f(x_i),y_i),\qquad R(f)=\mathbb{E}_{(X,Y)\sim P}[\ell(f(X),Y)]

Minimizing R^\widehat{R} is empirical risk minimization. It does not guarantee a small RR. Confidence comes from the combination of an appropriate inductive bias, sufficient coverage, representative sampling, and measurements on genuinely new cases. Interpolation within well-covered conditions and extrapolation beyond observed conditions make different demands on those assumptions.

What each score can support

EvidenceWhat it measuresWhat it does not establish
Training errorFit on examples used for learningPerformance on unseen cases
Random held-out rowsPerformance on similar held-out recordsGeneralization to new entities if related records cross the split
Held-out projectsPerformance on projects absent from fittingFuture performance after conditions change
Earlier-to-later splitForecast performance across the represented time boundaryStability beyond that period

Rows are not always independent evidence

Random shuffling does not create independence. A thousand builds from one repository may contain less evidence about a new repository than a hundred builds from a hundred repositories. Group-aware splitting keeps dependent observations together when the claim concerns new entities; chronological splitting answers a different question about future observations.

Validation without leakage

Split names describe responsibilities. The training set fits model parameters and fitted preprocessing. The validation set guides choices such as model family, hyperparameters, features, and stopping point. The test set is held aside until those choices are frozen, then estimates the selected system’s performance.

Selection can overfit too

Repeatedly changing the system after reading test results makes the test set part of development, even when its rows never enter a gradient calculation. Adaptive selection depends on the reused data through the reported scores. Specialized reusable-holdout procedures exist under explicit query and sampling conditions, but ordinary final-test claims require fresh, independently sampled cases after adaptive development.

Data leakage occurs when fitting or selection uses information that would be unavailable in the intended prediction setting. Common paths include scaling features with statistics computed from all records, selecting features before splitting, placing records from the same entity on both sides of a split, or using observations from the future. A pipeline can ensure that fitted transformations are learned within each training fold, but it cannot prove that the raw fields were genuinely available at prediction time.

Three data boundaries, three responsibilities

The test estimate remains independent only when the complete system is frozen before test outcomes are read.

Training dataContains only information permitted inside the fitting boundary.
Fitted candidatePreprocessing state and model parameters are learned from training data.
Validation assessmentEvaluate the fitted candidate on validation data; results guide selection, not ordinary parameter fitting.
Development choiceValidation results guide model family, features, hyperparameters, and stopping.
Frozen pipelineThe selected preprocessing and predictor after development decisions are complete.
Untouched test assessmentEvaluate the frozen pipeline on untouched test cases matching the final population or time claim.
Final estimatePerformance of the frozen pipeline on the represented population or time boundary.
Boundary rulesFit preprocessing inside training; keep claim-relevant groups together; exclude future information; obtain fresh cases after test-driven revisions.
Training data fit preprocessing and parameters; validation results guide choices; the frozen pipeline is assessed on untouched test data. All-data preprocessing, test-driven revision, future information, and related records crossing a claim-relevant group boundary violate this separation. A pipeline does not establish raw-field timing or group independence by itself.
Read the diagram as text
  • Training data. Contains only information permitted inside the fitting boundary.
  • Fitted candidate. Preprocessing state and model parameters are learned from training data.
  • Validation assessment. Evaluate the fitted candidate on validation data; results guide selection, not ordinary parameter fitting.
  • Development choice. Validation results guide model family, features, hyperparameters, and stopping.
  • Frozen pipeline. The selected preprocessing and predictor after development decisions are complete.
  • Untouched test assessment. Evaluate the frozen pipeline on untouched test cases matching the final population or time claim.
  • Final estimate. Performance of the frozen pipeline on the represented population or time boundary.
  • Boundary rules. Fit preprocessing inside training; keep claim-relevant groups together; exclude future information; obtain fresh cases after test-driven revisions.
  • Training data → Fitted candidate: fits state.
  • Fitted candidate → Validation assessment: evaluated for selection.
  • Validation assessment → Development choice: selection results.
  • Development choice → Frozen pipeline: freezes complete system.
  • Frozen pipeline → Untouched test assessment: one final assessment.
  • Untouched test assessment → Final estimate: scored outcomes.
  • Boundary rules → Training data: constrains fitting sample.
  • Boundary rules → Untouched test assessment: constrains assessment sample.

Match the boundary to the claim

Intended claimUseful boundaryReason
New independent cases from the same populationRandom holdout or cross-validationEstimates variation across independently sampled cases
New customers, repositories, or patientsGroup-aware splitPrevents related records from appearing in fitting and evaluation
Future requestsChronological splitPrevents later information from influencing earlier forecasts
Final performance after model selectionUntouched test setSeparates selection from assessment

Baselines and uncertainty

Always compare against a simple baseline. A regression baseline might predict the training-set mean for every request; a classifier might always predict the most frequent training class. Beating it does not prove deployment value, but failing to beat it reveals that the learned features have not justified their complexity. Evaluation proportions also have sampling uncertainty, especially with small samples or rare failures. These foundations lead into the broader system-level methods in Evals and Benchmarks.

Underfitting, overfitting, and regularization

Underfitting means the fitted system has not captured useful structure available for the task. Overfitting means improving fit increasingly reflects sample-specific patterns that do not transfer. High training error can result from an unsuitable model family or unsuccessful optimization. Low training error combined with worse validation performance suggests a generalization gap, but only after leakage and validation mismatch have been ruled out.

Constrain the fit, not the evidence

Regularization changes which fitted solutions are favored. An L2 penalty discourages large coefficients; early stopping selects a training point using validation performance. These are different interventions, and neither repairs invalid labels, future leakage, or an evaluation population that does not match deployment.

ObservationPlausible limitationNext controlled comparison
Training and validation performance are both poorLimited fit, weak features, or optimization failureTry a more suitable representation, model family, or verified optimization change
Training improves while validation worsensPossible overfitting or selection biasCheck leakage, then compare regularization, earlier stopping, or simpler choices
Validation improves as independent training examples increaseEvidence that representative data helpCollect more examples from the same intended population
Both curves remain poor as sample size growsPersistent task, feature, label, or family limitationRevisit the prediction contract rather than merely adding rows

Training-step curves and sample-size learning curves answer different questions. The former vary continued optimization with a fixed dataset. The latter vary how many training examples are available. More parameters do not imply a universal U-shaped generalization curve, and more data from the same biased source may simply reinforce the same shortcut.

Part IV — Uncertainty, decisions, and change

Sources of predictive uncertainty

A point prediction gives one estimated outcome. A predictive distribution assigns probabilities across possible outcomes, while a prediction interval describes a range intended to contain a future observation at a stated long-run rate under its assumptions. These are richer claims than a single duration estimate.

Variation versus limited knowledge

Aleatoric uncertainty is unresolved outcome variation given the available information. Two requests with the same recorded features may still take different times because relevant causes were not observed or because outcomes genuinely vary. Epistemic uncertainty is limited knowledge about the predictive relationship, often exposed where examples are sparse or the model family is inadequate. The distinction is relative to the information and model: adding an informative feature can explain variation that previously looked irreducible.

More representative examples can reduce uncertainty about the fitted relationship without making each future outcome predictable. Conversely, richer observations may reduce outcome ambiguity while leaving the relationship poorly estimated in a new region. Some internal uncertainty decompositions are not uniquely identified by the same predictive distribution, so named components should not be treated as directly observed quantities in every model.

Different intervals answer different questions

OutputQuestion
Point predictionWhat single estimate does the model produce?
Prediction intervalWhere might one new outcome fall?
Confidence interval for a meanHow uncertain is the estimated average response?

A number between zero and one is not automatically a validated probability. Token log probabilities describe the model’s generation distribution, not calibrated factual confidence. A generated confidence statement and agreement among several similar models require their own evaluation. Calibration can also deteriorate under shift even when it looked acceptable on the original validation population.

From probabilities to actions

A probabilistic classifier is calibrated in an evaluated population when cases assigned probability near pp experience the outcome about a fraction pp of the time. Calibration is a frequency property across comparable cases, not a guarantee for one case. It also differs from discrimination, the ability to rank higher-risk cases above lower-risk ones. Predicting the population base rate for everyone can be calibrated while ranking nobody.

An action requires consequences as well as probabilities. If a false positive costs CFPC_{FP}, a false negative costs CFNC_{FN}, and correct decisions have zero cost, choosing the positive action minimizes expected cost when:

p≥CFPCFP+CFNp\geq \frac{C_{FP}}{C_{FP}+C_{FN}}

The same probability can justify different actions

Predicted timeout probabilityFalse alarm costMissed-timeout costResult
0.30LowHighIntervene at a relatively low threshold
0.30HighLowDo not intervene at that threshold
0.30ModerateModerate, with useful human review availableDefer if the review path has acceptable capacity and delay

Rare events produce surprising alert streams

Base rates matter. In a constructed set of 10,000 cases containing 100 real events, detecting 80% yields 80 true alerts. A 1% false-positive rate among 9,900 non-events adds 99 false alerts. Only 80 of 179 alerts are real even though 1% sounds small. This arithmetic is not a measured deployment result; it exposes why event prevalence and the denominator must accompany an alert rate.

Abstention adds a reject or defer action. Coverage is the fraction accepted automatically; selective risk is error among accepted cases. Raising a threshold helps only when the score meaningfully orders mistakes, and lower coverage transfers work elsewhere. Evaluate that downstream path rather than treating abstention as free safety. In bug-review systems, long lists of invalid reports illustrate the operational cost of a prediction stream whose report volume overwhelms human triage.

When deployment conditions shift

Distribution shift means that development or evaluation conditions differ from use conditions. The phrase is broad; diagnosing the changed relationship matters. Under covariate shift, input frequencies P(X)P(X) change while P(Y∣X)P(Y\mid X) is assumed stable. Under label shift, outcome prevalence P(Y)P(Y) changes while P(X∣Y)P(X\mid Y) is assumed stable. Under concept shift, the relationship P(Y∣X)P(Y\mid X) itself changes.

Different changes require different evidence

Suppose large requests become more common but the duration relationship for each request size remains unchanged. That is compatible with covariate shift. If a service migration makes requests of the same size take different times, the conditional relationship has changed. The first problem may call for reweighting or evaluation on the new mix under strong assumptions; the second requires new labels and renewed validation of the predictive relationship.

A shortcut is a decision rule that works under ordinary development conditions but fails under a targeted change. Data, architecture, and training choices influence which shortcut a model finds. Held-out success drawn from the same shortcut-supporting conditions therefore does not prove that the intended rule was learned.

Detecting changed inputs and establishing degraded predictions are separate tasks. One study reported a controlled image shift that a detector noticed without harming the tested classifier. Conversely, an input-only test does not directly test a changed input–outcome relationship. Performance monitoring needs fresh outcomes, not merely feature drift alarms.

The model can change what it sees

Sequential systems add feedback. In imitation learning, a policy’s early mistake can move it into states absent from expert demonstrations; later decisions then face a new, policy-induced distribution. DAgger addresses this setting by executing the learned policy and collecting expert actions for states it actually encounters. The mechanism illustrates why deployment behavior can change future inputs. Longer-term update, retention, and forgetting strategies belong in Continual Learning.

Part V — Prediction is not intervention

Association, action, and causal effect

Prediction estimates an outcome under patterns represented in observed data. A causal effect compares outcomes under alternative interventions. The difference matters whenever a model output is used to choose an action: a variable can predict an outcome without being a useful lever for changing it.

A common cause can reverse the lesson

Imagine autoscaling records in which high load causes operators to add replicas and also increases latency. Replica count may therefore be positively associated with latency even if adding a replica under comparable load reduces latency. Load is a confounder because it influences both the observed action and the outcome. A predictor can learn the association accurately and still answer the intervention question incorrectly.

Load confounds the observed replica–latency relationship

Example

Ordinary scaling records combine a load-driven association with the effect of changing replica count.

Operational loadA common cause of the scaling decision and latency.
Replica countThe action whose effect is being studied.
LatencyThe outcome affected by load and potentially by replica count.
The arrows are an assumed causal structure for this invented example. Observed association mixes the load-driven path with the replica effect. Under randomized assignment, replace the load-to-replica selection mechanism with random assignment; retain load → latency and replica count → latency. This supports an average-effect comparison under the experimental conditions, not both outcomes for one request.
Read the diagram as text
  • Operational load. A common cause of the scaling decision and latency.
  • Replica count. The action whose effect is being studied.
  • Latency. The outcome affected by load and potentially by replica count.
  • Operational load → Replica count: ordinary scaling responds to load.
  • Operational load → Latency: load affects outcome.
  • Replica count → Latency: possible action effect.

Editing the replica-count feature and rescoring a predictive model is not automatically a causal experiment. Feature attribution assigns parts of a prediction to inputs under a reference construction; it does not establish the real-world effect of changing the corresponding quantity. Causal reasoning needs a specified intervention and assumptions about how outcomes would be generated under it.

What intervention evidence requires

Random assignment makes action selection independent of potential outcomes in expectation, allowing differences between assigned groups to estimate an average effect under the experiment’s conditions. It does not reveal both possible outcomes for one request. Observational adjustment instead needs assumptions such as consistency, conditional exchangeability, and positivity: the action must be well specified, measured covariates must remove action–outcome confounding, and each relevant group must have support for the actions being compared.

Prediction creates feedback

Acting on predictions can also change later data. Prices, traffic, reviewer decisions, and alerts can alter the inputs and labels collected for the next model. An interface that encourages reviewers to approve model suggestions can feed model-influenced decisions back as apparent truth; an independent judgment step can reveal disagreements, though it does not guarantee correct labels.

Four claims, four evidence burdens

ClaimEvidence needed
The model forecasts accuratelyIndependent, representative outcomes scored with a defined metric
The forecast supports a useful decisionValidated probabilities or rankings plus consequences, capacity, and alternatives
The action changes the outcomeRandomized evidence or observational identification with defensible causal assumptions
The complete system helps in operationEnd-to-end evaluation of outcomes, side effects, adoption, and changed conditions

Open questions

  1. How can useful epistemic uncertainty be estimated when several internal uncertainty representations produce the same predictive distribution? Progress would require methods whose uncertainty distinctions are identifiable, empirically connected to future errors, and validated after realistic distribution changes.

  2. How should adaptive development reuse valuable failures without converting every benchmark into training data? The challenge is preserving regression tests while obtaining an honest estimate of performance on the intended population. Progress would combine versioned development sets, fresh independent cases, and reusable-holdout methods whose sampling and query-budget assumptions are actually enforced.

  3. How can teams measure the causal benefit of model-guided actions when deployment changes who receives an action and which outcomes become observable? Progress would require explicit intervention definitions, support for credible comparisons, measurement of feedback effects, and evaluation beyond predictive accuracy.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

268 matching talks

TalkSpeakerEventYear
Brendan RappazzoAI Engineer World's Fair 20262026
Diane LinAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Merve NoyanAI Engineer Europe 20262026
Ilan BigioAI Engineer World's Fair 20252025
Vibhor KumarAI Engineer World's Fair 20242024
Eugene YanAI Engineer Summit 20232023
Angelos PerivolaropoulosAI Engineer Europe 20262026
Zhengyao JiangAI Engineer World's Fair 20262026
Isaac RobinsonAI Engineer Europe 20262026
Sina ShahandehAI Engineer World's Fair 20262026
Phil HetzelAI Engineer Europe 20262026
Shafik QuoraisheeAI Engineer World's Fair 20252025
Ben HylakAI Engineer World's Fair 20262026
Daniel HanAI Engineer World's Fair 20252025
Stefano FiorucciAI Engineer Europe 20262026
Gaurav MishraAI Engineer World's Fair 20262026
Robotics: why now?

Transcript reviewed

Quan Vuong, Jost Tobias SpringenbergAI Engineer World's Fair 20252025
Preetika Bhateja, Daniel BumpAI Engineer World's Fair 20262026
Low Level Technicals of LLMs

Transcript reviewed

Daniel HanAI Engineer World's Fair 20242024
Louis-François Bouchard, Paul Iusztin, Samridhi VaidAI Engineer Europe 20262026
Jesse HuAI Engineer Code 20252025
Parth AsawaAI Engineer World's Fair 20262026
Justin ReockAI Engineer Code 20252025
LLM Evals That Work IRL

Transcript reviewed

Aparna Dhinkaran, Aparna DhinakaranAI Engineer World's Fair 20242024
Ben KunkleAI Engineer Europe 20262026
Tomas ReimersAI Engineer World's Fair 20252025
#define AI Engineer

Transcript reviewed

Greg Brockman, swyx, Jensen HuangAI Engineer World's Fair 20252025
Kelvin MaAI Engineer World's Fair 20252025
Alex Shaw, Ryan MartenAI Engineer World's Fair 20262026
Yesu FengAI Engineer World's Fair 20252025
What's next after RLHF?

Transcript reviewed

Diogo AlmeidaAI Engineer World's Fair 20262026
Mohak SharmaAI Engineer Summit 20252025
Anna Marie BenzonAI Engineer World's Fair 20262026
Ian ButlerAI Engineer World's Fair 20252025
Akele Reed, Dave Revere, Doug KellerAI Engineer World's Fair 20262026
Evaling Video Slop

Transcript reviewed

Maor BrilAI Engineer World's Fair 20262026
Hamza TahirAI Engineer World's Fair 20262026
Udi MenkesAI Engineer World's Fair 20262026
Nathaniel Whittemore (NLW)AI Engineer Code 20252025
Ara KhanAI Engineer Europe 20262026
Ari MorcosAI Engineer World's Fair 20262026
Ian Butler, Nick GregoryAI Engineer World's Fair 20252025
Bertrand CharpentierAI Engineer Europe 20262026
John DickersonAI Engineer World's Fair 20252025
Justin MullerAI Engineer World's Fair 20252025
Nathan LambertAI Engineer World's Fair 20252025
Logan KilpatrickAI Engineer World's Fair 20252025
Sharmila Chokalingam, ShubhiAI Engineer World's Fair 20242024
Chintan Parikh, Weiyi WangAI Engineer Europe 20262026
Sara HookerAI Engineer World's Fair 20262026
Jacob LauritzenAI Engineer Europe 20262026
Tejas KumarAI Engineer Europe 20262026
Dax RaadAI Engineer Code 20252025
Nagkumar Arkalgud, Keiji KanazawaAI Engineer World's Fair 20252025
Benjamin DunphyAI Engineer Summit 20232023
Anthropic for VPs of AI

Metadata candidate

Alexander Bricken, Joe BayleyAI Engineer Summit 20252025
Frank CoyleAI Engineer World's Fair 20262026
Richmond AlakeAI Engineer World's Fair 20252025
Ivan BurazinAI Engineer World's Fair 20252025
Grace IsfordAI Engineer Summit 20252025
Łukasz GandeckiAI Engineer World's Fair 20252025
Aparna DhinakaranAI Engineer World's Fair 20252025
Sunny MadraAI Engineer World's Fair 20242024
Greg BensonAI Engineer World's Fair 20252025
Eliza Cabrera, Jeremy SilvaAI Engineer World's Fair 20252025
Varun Badrinath Krishna, Petro Junior Milan, Rachelle MatternAI Engineer World's Fair 20242024
Cedric Vidal, David Smith, Miguel MartinezAI Engineer World's Fair 20242024
Max Brodeur-UrbasAI Engineer World's Fair 20252025
Building a Chess Coach

Metadata candidate

Anant Dole, Asbjørn SteinskogAI Engineer Europe 20262026
Du’An Lightfoot, Banjo ObayomiAI Engineer World's Fair 20252025
Bruno Passos, Beyang LiuAI Engineer Summit 20252025
Ben Hylak, Sid BendreAI Engineer World's Fair 20252025
Ben KusAI Engineer World's Fair 20252025
Angie JonesAI Engineer World's Fair 20262026
Cedric VidalAI Engineer World's Fair 20252025
Anoop Kotha, Toki SherbakovAI Engineer World's Fair 20252025
Matt PocockAI Engineer World's Fair 20262026
Adam TerlsonAI Engineer Summit 20252025
Apoorva JoshiAI Engineer World's Fair 20252025
Building Reactive AI Apps

Metadata candidate

Matt WelshAI Engineer Summit 20232023
Eno ReyesAI Engineer World's Fair 20242024
Sandra KublikAI Engineer World's Fair 20242024
Liam McGarrigleAI Engineer Europe 20262026
Andrew ThompsonAI Engineer World's Fair 20252025
Anant ShankhdharAI Engineer World's Fair 20262026
Anju KambadurAI Engineer Summit 20252025
Ben HolmesAI Engineer World's Fair 20252025
Derek BinghamAI Engineer World's Fair 20242024
Amelia WattenbergerAI Engineer Summit 20232023
Codex and Subagents

Metadata candidate

Vaibhav Srivastav, Katia Gil GuzmanAI Engineer Europe 20262026
Cohere for VPs of AI

Metadata candidate

Vivek MuppallaAI Engineer World's Fair 20242024
Yusuf OlokobaAI Engineer Code 20252025
Priscila Andre de OliveiraAI Engineer Europe 20262026
Compression at the Edge

Metadata candidate

Chris Alexiuk, Daniel Han, Asma Beevi, Merve Noyan, Parth SareenAI Engineer World's Fair 20262026
Louis-François Bouchard, Omar Solano, Samridhi VaidAI Engineer World's Fair 20262026
Convex Launch

Metadata candidate

Jamie TurnerAI Engineer World's Fair 20242024
Karina NguyenAI Engineer Summit 20252025
Vincent KocAI Engineer Europe 20262026
Mahesh SathiamoorthyAI Engineer World's Fair 20262026
Mani KhanujaAI Engineer World's Fair 20252025
Defying Gravity

Metadata candidate

Kevin HouAI Engineer Code 20252025
Zeke SikelianosAI Engineer World's Fair 20252025
Chintan Agrawal, Daniel WirjoAI Engineer World's Fair 20262026
Develop at Idea Velocity

Metadata candidate

Jeffrey Lee-ChanAI Engineer World's Fair 20262026
Phil HetzelAI Engineer Europe 20262026
Abi AryanAI Engineer Summit 20232023
Ara KhanAI Engineer Europe 20262026
Laurie VossAI Engineer World's Fair 20252025
Satya NittaAI Engineer World's Fair 20242024
Aparna Dhinkaran, Aparna DhinakaranAI Engineer Summit 20252025
Garry TanAI Engineer World's Fair 20262026
Maxime LabonneAI Engineer Europe 20262026
Maxime LabonneAI Engineer World's Fair 20242024
Field Guide to Fable

Metadata candidate

Thariq ShihiparAI Engineer World's Fair 20262026
Daniel HanAI Engineer World's Fair 20242024
Cormac BrickAI Engineer Europe 20262026
Samuel ColvinAI Engineer Code 20252025
Alex CheemaAI Engineer Europe 20262026
Rachel Lee Nabors (RL Nabors)AI Engineer World's Fair 20262026
Jason LiuAI Engineer World's Fair 20262026
Alex AtallahAI Engineer World's Fair 20252025
Fuzzing in the GenAI Era

Metadata candidate

Leonard TangAI Engineer World's Fair 20252025
Omar SansevieroAI Engineer Europe 20262026
Phoebe KlettAI Engineer World's Fair 20242024
Matija SosicAI Engineer Summit 20232023
Audry HsuAI Engineer Europe 20262026
How Deep Research Works

Metadata candidate

Mukund Sridhar, Aarush SelvanAI Engineer Summit 20252025
Vinesh GudlaAI Engineer World's Fair 20252025
Jaspreet SinghAI Engineer World's Fair 20252025
Benjamin VerbeekAI Engineer Europe 20262026
Patrick DoughertyAI Engineer Summit 20252025
Yogendra MirajeAI Engineer World's Fair 20252025
Zhou YuAI Engineer Summit 20252025
Kyle CorbittAI Engineer World's Fair 20252025
HTML Is All Agents Need

Metadata candidate

James RussoAI Engineer World's Fair 20262026
Amol KapoorAI Engineer World's Fair 20262026
Nicolas SchlaepferAI Engineer World's Fair 20242024
Radek SienkiewiczAI Engineer Europe 20262026
Suman DebnathAI Engineer World's Fair 20252025
Sarthak AggarwalAI Engineer World's Fair 20262026
Ian WebsterAI Engineer World's Fair 20242024
Mahmoud MabroukAI Engineer Europe 20262026
Judging LLMs

Metadata candidate

Alex VolkovAI Engineer World's Fair 20242024
Chip HuyenAI Engineer Summit 20252025
David KaramAI Engineer World's Fair 20252025
Divakar KumarAI Engineer World's Fair 20262026
Rukma SenAI Engineer World's Fair 20242024
Stephen Hood, Justine TunneyAI Engineer World's Fair 20242024
Danilo CamposAI Engineer Europe 20262026
Ben HolmesAI Engineer World's Fair 20262026
2025 in LLMs so far

Metadata candidate

Simon WillisonAI Engineer World's Fair 20252025
Thierry Moreau, Pedro TorruellaAI Engineer World's Fair 20242024
Hubert MisztelaAI Engineer World's Fair 20242024
Carter Abdallah, Vincent Weisser, Lucas Atkins, Chris AlexiukAI Engineer World's Fair 20262026
Joe FiotiAI Engineer World's Fair 20252025
Theodora ChuAI Engineer World's Fair 20252025
Manuel OdendahlAI Engineer World's Fair 20252025
Vikhyat KorrapatiAI Engineer World's Fair 20242024
Martin Harrysson, Natasha ManiarAI Engineer Code 20252025
Rami AlhamadAI Engineer World's Fair 20252025
Ahmed MenshawyAI Engineer World's Fair 20242024
Rémi LoufAI Engineer World's Fair 20242024
Notion's Token Town

Metadata candidate

Sarah SachsAI Engineer World's Fair 20262026
Sharif ShameemAI Engineer World's Fair 20252025
Frank CoyleAI Engineer World's Fair 20262026
Simon WillisonAI Engineer Summit 20232023
Ofer MendelevitchAI Engineer World's Fair 20252025
DottaAI Engineer Europe 20262026
Antje BarthAI Engineer World's Fair 20262026
Ishan AnandAI Engineer World's Fair 20262026
Shivam VermaAI Engineer World's Fair 20252025
Mario ZechnerAI Engineer Europe 20262026
Anish Agarwal, Matthew SchoenbauerAI Engineer World's Fair 20252025
Lukas BiewaldAI Engineer World's Fair 20242024
Jason LiuAI Engineer World's Fair 20242024
Yuval BelferAI Engineer World's Fair 20252025
Recursive Coding Agents

Metadata candidate

Raymond WeitekampAI Engineer World's Fair 20262026
Recursive Model Improvement

Metadata candidate

Lee RobinsonAI Engineer World's Fair 20262026
Benoit SchillingsAI Engineer World's Fair 20262026
Rayan GargAI Engineer World's Fair 20262026
RL Environments at Scale

Metadata candidate

Will BrownAI Engineer Code 20252025
Stephan SteinfurtAI Engineer Europe 20262026
Pamela Fox, Harald Kirschner, Gabriela de QueirozAI Engineer World's Fair 20242024
Gabriela de Queiroz, Aishwarya Srinivasan, Pamela FoxAI Engineer World's Fair 20242024
Adrien GrondinAI Engineer Europe 20262026
Scaling Compute on Context

Metadata candidate

Jack MorrisAI Engineer World's Fair 20262026
Mike ChambersAI Engineer World's Fair 20252025
Eno ReyesAI Engineer World's Fair 20252025
Aman KhanAI Engineer World's Fair 20252025
Giran Moodley, Mayan Soni, Oussama Hafferssas, Mayank SoniAI Engineer Europe 20262026
Gus Martins, Ian BallantyneAI Engineer Europe 20262026
Steven WillmottAI Engineer Europe 20262026
Daniel HanAI Engineer World's Fair 20262026
Ishan AnandAI Engineer World's Fair 20242024
Sarah GuoAI Engineer World's Fair 20252025
Peter Steinberger, swyxAI Engineer Europe 20262026
Alejandro VidalAI Engineer World's Fair 20262026
Kobie CrawfordAI Engineer Europe 20262026
Jan SimlAI Engineer World's Fair 20252025
Pydantic is all you need

Metadata candidate

Jason LiuAI Engineer Summit 20232023
Denys LinkovAI Engineer World's Fair 20252025
Jack MorrisAI Engineer Code 20252025
Kobie CrawfordAI Engineer Europe 20262026
Brendan O'DonoghueAI Engineer Europe 20262026
Barr YaronAI Engineer World's Fair 20262026
Patrick DeboisAI Engineer World's Fair 20252025
Sumit AgarwalAI Engineer World's Fair 20242024
Jack CableAI Engineer World's Fair 20262026
Chris White, Bryan Bischof, Brittany WalkerAI Engineer Summit 20232023
The Base Model is Dead

Metadata candidate

Varun SinghAI Engineer World's Fair 20262026
Kevin Madura, Mo BhasinAI Engineer World's Fair 20252025
Maximillian PirasAI Engineer World's Fair 20252025
Jeremy Silva, Chris HernandezAI Engineer World's Fair 20252025
Travis FrisingerAI Engineer World's Fair 20252025
Corey J. GallonAI Engineer Code 20252025
Kieran KlaassenAI Engineer World's Fair 20262026
Justin SchroederAI Engineer World's Fair 20262026
Ankur GoyalAI Engineer World's Fair 20252025
Aparna DhinakaranAI Engineer World's Fair 20262026
Junyang LinAI Engineer World's Fair 20252025
The Future of Work

Metadata candidate

Toran Bruce Richards, Silen Naihin, Reinier van der LeerAI Engineer Summit 20232023
Kyle CorbittAI Engineer World's Fair 20242024
Alexander Embiricos, Romain Huet, Peter SteinbergerAI Engineer World's Fair 20262026
The Intelligent Interface

Metadata candidate

Samantha Whitmore, Jason YuanAI Engineer Summit 20232023
Almog BakuAI Engineer Summit 20252025
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
Hassan El MghariAI Engineer World's Fair 20262026
The New Lean Startup

Metadata candidate

Sid BendreAI Engineer World's Fair 20252025
Kshitij GroverAI Engineer Summit 20252025
Amir HaghighatAI Engineer World's Fair 20252025
Elmer Thomas, Maria BermudezAI Engineer World's Fair 20252025
Walden Yan, Carter Abdallah, Tanay Varshney, Alex Atallah, Nader KhalilAI Engineer World's Fair 20262026
The Weekend AI Engineer

Metadata candidate

Hassan El MghariAI Engineer Summit 20232023
Danielle An, David HoeAI Engineer Europe 20262026
Thinking Deeper in Gemini

Metadata candidate

Jack RaeAI Engineer World's Fair 20252025
Forrest Brazeal, Matt BallAI Engineer World's Fair 20252025
Leo PekelisAI Engineer World's Fair 20242024
Micah Hill-Smith, George CameronAI Engineer World's Fair 20252025
Paul Iusztin, Louis-François BouchardAI Engineer World's Fair 20262026
Eric AllamAI Engineer World's Fair 20252025
Thabang LedwabaAI Engineer World's Fair 20252025
Jon PeckAI Engineer World's Fair 20252025
Kathleen KenealyAI Engineer World's Fair 20242024
Hassan El MghariAI Engineer World's Fair 20252025
Victor DibiaAI Engineer World's Fair 20252025
Matt DaileyAI Engineer World's Fair 20262026
Peter GostevAI Engineer Europe 20262026
Charles FryeAI Engineer World's Fair 20252025
Arjun Bansal, Trey DoigAI Engineer World's Fair 20242024
Alex AlbertAI Engineer World's Fair 20242024
Eugene Yan, Hamel Husain, Jason Liu, Dr Bryan Bischof, Charles Frye, Shreya ShankarAI Engineer World's Fair 20242024
Mukuntha Narayanan, Han WangAI Engineer World's Fair 20252025
Andy TriedmanAI Engineer Summit 20252025
Tom Shapland, PhDAI Engineer World's Fair 20252025
Nupur SharmaAI Engineer Europe 20262026
Mark BissellAI Engineer World's Fair 20252025
Chin Keong LamAI Engineer World's Fair 20252025
Karina NguyenAI Engineer Summit 20232023
Eugene CheahAI Engineer Summit 20252025
Erik HanchettAI Engineer World's Fair 20262026
AI Engineer Summit 20252025
Edo LibertyAI Engineer World's Fair 20262026
Varun KrovvidiEvent unknownUnknown
Somya Rai, MD ShariqueAI Engineer NYC 2026 Online Talks2026
Zixuan LiEvent unknownUnknown
Hrishi OlickelAI Engineer NYC 2026 Online Talks2026
Yuxuan ZhangAI Engineer Code 20252025

References

Coverage and source review
Processed transcripts
46 processed in full · 3 in the curated path
Automated source review
Passed
Metadata candidates
225 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Dive into Deep Learning: Generalization in Deep Learning

    https://en.d2l.ai/chapter_multilayer-perceptrons/generalization-deep.html

  2. PyTorch Quickstart: forward computation, training, and loading

    https://docs.pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html

  3. PyTorch 2.9: Reproducibility

    https://docs.pytorch.org/docs/2.9/notes/randomness.html

  4. Google: Supervised learning foundations

    https://developers.google.com/machine-learning/intro-to-ml/supervised

  5. Dive into Deep Learning: Linear Regression

    https://classic.d2l.ai/chapter_linear-networks/linear-regression.html

  6. Scikit-learn: scoring rules and baseline estimators

    https://scikit-learn.org/stable/modules/model_evaluation.html

  7. scikit-learn: Preprocessing data

    https://scikit-learn.org/stable/modules/preprocessing.html

  8. TravisTorrent: Data Format

    https://testroots.github.io/travistorrent-site/page_dataformat/

  9. scikit-learn: Common pitfalls and recommended practices

    https://scikit-learn.org/stable/common_pitfalls.html

  10. The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables

    https://cs.stanford.edu/people/jure/pubs/contraction-kdd17.pdf

  11. Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse

    https://www.youtube.com/watch?v=eAXxdtNlK04&t=181

  12. L2 regularization and generalization

    https://developers.google.com/machine-learning/crash-course/overfitting/regularization

  13. scikit-learn: Decision Trees

    https://scikit-learn.org/stable/modules/tree.html

  14. Rumelhart, Hinton and Williams: Learning Representations by Back-Propagating Errors

    https://gwern.net/doc/ai/nn/1986-rumelhart-2.pdf

  15. Geirhos et al.: Shortcut Learning in Deep Neural Networks

    https://arxiv.org/abs/2004.07780

  16. Legendre: Nouvelles méthodes pour la détermination des orbites des comètes

    https://www.e-rara.ch/zut/download/ftpack/plain/10838079

  17. Rosenblatt: The Perceptron—A Probabilistic Model for Information Storage and Organization in the Brain

    https://web.stanford.edu/class/psych209a/ReadingsByDate/01_30/Rosenblatt58Perceptron.pdf

  18. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    https://aclanthology.org/N19-1423.pdf

  19. How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed

    https://www.youtube.com/watch?v=phchDt63qAA&t=83

  20. scikit-learn: Clustering

    https://scikit-learn.org/stable/modules/clustering.html

  21. Reinforcement Learning: An Introduction — second-edition draft

    https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf

  22. scikit-learn: Mean absolute error

    https://scikit-learn.org/stable/modules/model_evaluation.html

  23. Logistic regression: binary log loss

    https://developers.google.com/machine-learning/crash-course/logistic-regression/loss-regularization

  24. Training language models to follow instructions with human feedback

    https://arxiv.org/pdf/2203.02155

  25. Weak supervision needs evaluation beyond supervisor agreement

    https://arxiv.org/pdf/2312.09390

  26. Minibatch Stochastic Gradient Descent — Dive into Deep Learning

    https://d2l.ai/chapter_optimization/minibatch-sgd.html

  27. PyTorch: Optimizing Model Parameters

    https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html

  28. Training an LLM from Scratch, Locally

    https://www.youtube.com/watch?v=UsB70Tf5zcE&t=2717

  29. Robbins and Monro: A Stochastic Approximation Method

    https://www.columbia.edu/~ww2040/8100F16/RM51.pdf

  30. Language Models are Few-Shot Learners

    https://arxiv.org/html/2005.14165v4

  31. Dive into Deep Learning: generalization in classification

    https://d2l.ai/chapter_linear-classification/generalization-classification.html

  32. NIST/SEMATECH e-Handbook: How do we Use the Model Beyond the Data Domain?

    https://www.itl.nist.gov/div898/handbook/pri/section5/pri5998.htm

  33. scikit-learn: Cross-validation for grouped and time-series data

    https://scikit-learn.org/stable/modules/cross_validation.html

  34. Google ML Crash Course: Dividing Datasets

    https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets

  35. Generalization in Adaptive Data Analysis and Holdout Reuse

    https://arxiv.org/html/1506.02629

  36. Cawley and Talbot: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation

    https://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley10a.pdf

  37. Cross-validation and held-out evaluation

    https://scikit-learn.org/stable/modules/cross_validation.html

  38. scikit-learn: DummyRegressor

    https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html

  39. NIST: Confidence Intervals for a Proportion

    https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm

  40. Dive into Deep Learning: Model Selection, Underfitting, and Overfitting

    https://classic.d2l.ai/chapter_multilayer-perceptrons/underfit-overfit.html

  41. Hoerl and Kennard: Ridge Regression—Biased Estimation for Nonorthogonal Problems

    https://paperzz.com/doc/8629431/ridge-regression--biased-estimation-for

  42. scikit-learn: Validation Curves—Plotting Scores to Evaluate Models

    https://scikit-learn.org/stable/modules/learning_curve.html

  43. NIST: How Can I Predict the Value and Estimate the Uncertainty of a Single Response?

    https://www.itl.nist.gov/div898/handbook/pmd/section5/pmd512.htm

  44. Hüllermeier and Waegeman: Aleatoric and Epistemic Uncertainty in Machine Learning

    https://arxiv.org/html/1910.09457

  45. Jürgens et al.: Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?

    https://arxiv.org/html/2402.09056

  46. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=900

  47. Guo et al.: On Calibration of Modern Neural Networks

    https://proceedings.mlr.press/v70/guo17a/guo17a.pdf

  48. Ovadia et al.: Can You Trust Your Model's Uncertainty?

    https://papers.neurips.cc/paper_files/paper/2019/file/8558cb408c1d76621371888657d2eb1d-Paper.pdf

  49. Hébert-Johnson et al.: Multicalibration—Calibration for the Computationally-Identifiable Masses

    https://proceedings.mlr.press/v80/hebert-johnson18a/hebert-johnson18a.pdf

  50. Elkan: The Foundations of Cost-Sensitive Learning

    https://www.researchgate.net/publication/2365611_The_Foundations_of_Cost-Sensitive_Learning

  51. Axelsson: The Base-Rate Fallacy and the Difficulty of Intrusion Detection

    https://profsandhu.com/cs6393_s21/Axelsson-2000.pdf

  52. Geifman and El-Yaniv: Selective classification and risk control

    https://arxiv.org/pdf/1705.08500

  53. Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

    https://www.youtube.com/watch?v=wAQK7O3WGEE&t=866

  54. Dive into Deep Learning: environment and distribution shift

    https://en.d2l.ai/chapter_linear-classification/environment-and-distribution-shift.html

  55. Dive into Deep Learning: Concept Shift

    https://en.d2l.ai/chapter_linear-classification/environment-and-distribution-shift.html

  56. Rabanser, Günnemann and Lipton: Failing Loudly

    https://papers.neurips.cc/paper_files/paper/2019/file/846c260d715e5b854ffad5f70a516c88-Paper.pdf

  57. Ross, Gordon and Bagnell: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

    https://proceedings.mlr.press/v15/ross11a/ross11a.pdf

  58. Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

    https://www.youtube.com/watch?v=qqXdLf3wy1E&t=523

  59. Hernán and Robins: Causal Inference—What If

    https://www.popdata.bc.ca/sites/default/files/documents/events/etu/Hernan_Book.pdf

  60. When observational action records identify causal effects

    https://www.popdata.bc.ca/sites/default/files/documents/events/etu/Hernan_Book.pdf

  61. Feature attribution is not a real-world intervention effect

    https://proceedings.mlr.press/v108/janzing20a/janzing20a.pdf

  62. Holland: Statistics and Causal Inference

    https://www.cs.columbia.edu/~blei/fogm/2018F/materials/Holland1986.pdf

  63. Perdomo et al.: Performative Prediction

    https://proceedings.mlr.press/v119/perdomo20a/perdomo20a.pdf

  64. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    https://www.youtube.com/watch?v=CDqzWpwkSls&t=563

  65. Rolling-origin forecast evaluation

    https://otexts.com/fpp3/tscv.html

  66. How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI

    https://www.youtube.com/watch?v=iCj_ATyThvc&t=828

  67. Beyond Static Intelligence: Evaluating Continual Learning

    https://www.youtube.com/watch?v=iqloyWCGYQQ&t=689

  68. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    https://www.youtube.com/watch?v=ZuiJjkbX0Og&t=3857

  69. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    https://www.youtube.com/watch?v=ZuiJjkbX0Og&t=4213

  70. Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse

    https://www.youtube.com/watch?v=eAXxdtNlK04&t=296

  71. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    https://www.youtube.com/watch?v=CDqzWpwkSls&t=359