AI Engineer World's Fair 2025
Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai
Loading slides…
Read the talk
Perceptual Evaluations: Evals for Aesthetics
A malformed hand, perceptual compression and a fragile image metric expose the gap between measuring visual properties and judging whether an image works for a person.
From a talk by Diego Rodriguez
The hand a model almost accepts
An AI-generated hand has distorted fingers and joints. To a person, the anatomy looks immediately wrong. Diego Rodriguez, Krea’s co-founder, opens with a simple test: ask o3 what it thinks of the image. In Rodriguez’s account, o3 spends 17 seconds on a tool-assisted assessment involving Python and OpenCV. It notices melting joints, yet largely accepts the hand as natural. The elaborate inspection misses the force of the immediate human reaction.
This is one anecdote, not a benchmark of the model’s visual abilities. But it poses a useful evaluation problem: recognizing visual features is not the same as judging how an image appears to a person. Models learn from human-produced data and human preferences, while the data itself reflects what people can perceive and choose to record. Why does that training still leave such a gap in judgment?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Which information reaches the recipient?
Rodriguez approaches the problem through Claude Shannon. Shannon’s early work on switching circuits and his later communication theory are distinct contributions; the relevant paper here is A Mathematical Theory of Communication. Its framing matters: information travels from a source through a channel to a destination, with noise potentially entering along the way. The system exists to communicate something to a recipient.
For generative media, that framing connects representation to perception. Rodriguez draws a conceptual resemblance between communication diagrams and neural networks or variational autoencoders, rather than claiming they are equivalent systems. Compression supplies the more concrete connection: a representation can lose information in transit while preserving what matters to the recipient. That makes the recipient’s sensitivity part of the engineering problem—and, eventually, part of evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deleting information without changing what people see
JPEG makes this relationship tangible. Human vision is more sensitive to fine brightness detail than to fine color detail, so an image representation need not preserve both at the same resolution. Alongside this explanation, Rodriguez shows an illusion in which regions A and B have the same color but appear different until the reveal. The illusion illustrates how context affects appearance; the compression example exploits a particular difference in visual sensitivity.
The demonstrated sequence separates the image into components before reducing information:
- Separate brightness and color. Convert the RGB representation into components that distinguish brightness from color differences. The slide labels these
Y,CbandCr. - Downsample the color components. Reduce their spatial resolution while retaining the brightness detail to which vision is more sensitive.
- Reconstruct the image. Combine the components again and compare the result with the original.
The pixelated letter R makes the separation visible: the brightness component and the two color components carry different parts of the same image.
Rodriguez describes the displayed reconstruction as looking the same with roughly 50 percent less information. That figure belongs to this example, not to a universal JPEG file-size guarantee. The essential result is that information can be removed without producing a difference the viewer readily notices.
Other compression steps do different work. Huffman coding encodes symbols efficiently; it is distinct from deliberately discarding less noticeable visual information. Rodriguez then extends the perceptual principle to MP3: what can listeners hear, and what can be removed without a noticeable loss? Video adds opportunities to reduce redundancy across time. His MP4 analogy is shorthand: MP4 is a container format, while the audiovisual codecs inside it perform the compression. Across these examples, human perception helps determine which losses are acceptable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compression enters both the dataset and the metric
Studying audiovisual systems engineering made this limitation concrete for Rodriguez. He could delete information in code, render the image again and still see the same thing. Knowing that data had disappeared did not make the difference perceptible. The exercise turned an abstract claim about the limits of the senses into something he could directly experience.
Now move that observation into a training pipeline. Internet images have often already been compressed. Before a model encounters them, their representations contain decisions about what human viewers are unlikely to miss. Rodriguez asks whether training on that material passes some of our perceptual limitations into AI. It is a question about the consequences of data selection and representation, not a demonstrated causal result.
Evaluation introduces another mismatch. Rodriguez points to a JPEG example associated with clean-fid: four displayed images appear essentially alike to him, while the FID results react strongly to compression artifacts. Fréchet Inception Distance (FID) compares image distributions through image features; it is not an aesthetic rating of an individual picture. The distinction matters because a distributional difference can be real without being the difference a human evaluator cares about.
The lesson is not that JPEG must always worsen FID. Its effect depends on the reference distribution and the evaluation pipeline. Rather, a metric can be sensitive to changes that barely affect perceived quality. Before using such a score to select a generative model, the practical question is whether its sensitivity matches the judgment the selection is supposed to serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An incorrect clock can be successful art
It is tempting to evaluate what is easy to specify: prompt adherence using CLIP, object counts, or whether an object is blue or red. These are useful but limited targets. Image–text matching addresses correspondence between an image and language; counting and color checks address particular visible properties. None alone establishes whether an image succeeds aesthetically.
Rodriguez then turns to art with an unusual clock and sky. A literal realism check could treat both as generation failures. In an artwork, however, distortion may be intentional and central to the meaning. The viewer’s interpretation depends on what the author is communicating, not just whether the depicted objects conform to ordinary physical appearances. A departure from realism can be the point of the image.
That changes the objective for a creative tool. Krea’s purpose, as Rodriguez describes it, is to help artists and other creatives express themselves through imagery and video. An evaluation system that penalizes intentional departures from realism can work against that purpose. Expressive usefulness belongs in the definition of success, even when it is harder to specify than an object count.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From predicting cars to anticipating traffic
The discussion then widens from evaluating a capability to understanding its consequences. Rodriguez recalls an analogy from his friend Cheng Lu, whom he identifies as working at Midjourney: predicting the car in a world of horses is comparatively straightforward. Keep the vehicle, replace the horse’s energy with an engine, and the basic idea is there. Predicting traffic is harder. Traffic emerges when many people adopt and use the invention together.
What is the equivalent of traffic for AI? Rodriguez asks engineers, researchers and founders to look beyond isolated capabilities toward the changes those capabilities enable at scale. Converting JSON to YAML is his example of a narrow task that can absorb attention. He concedes that such tasks matter; the concern is that measuring them can leave the larger consequences unexplored.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Translation changes who can work together
The Tower of Babel supplies the next analogy. In Rodriguez’s retelling, incompatible languages prevent people from coordinating and completing their shared project. He compares that breakdown to infrastructure meetings where arguments over Kubernetes and other choices prevent anything from getting built. Communication failures can block collective action even when the participants have the necessary technical abilities.
Increasingly capable translation changes that constraint. Rodriguez describes translation as solved to a high degree, rather than perfectly solved, and asks what follows when people can speak their own languages while still communicating with one another. His example is already practical: he uses translation to provide customer support to Japanese Krea users despite not speaking Japanese fluently. It extends direct support from a founder to people he otherwise could not serve in that way.
The consequence reaches beyond exchanging factual instructions. Better communication also helps people convey opinions and understand one another’s preferences. That brings the discussion back to aesthetics: if a tool is meant to support expression, it needs to understand more of what a person means by a successful result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate what the evaluation captures
A cat image exposes several different questions that an evaluation might collapse into one:
| Criterion | Question |
|---|---|
| Count | Are there four cats? |
| Color description | Is this cat blue or teal? |
| Personal preference | Do I like this particular blue? |
A correct count does not settle the hue, and an accurate hue description does not establish whether the viewer likes it. Preference adds a target beyond descriptive accuracy.
Evaluating an evaluation means asking whether it captures the intended person’s judgment. Rodriguez wants metrics that account for his opinion, rather than assuming one score exhausts what makes an image good. His reference to being a visual learner expresses that personal perspective. The broader design requirement is to account for how people perceive images and for the nature of the data used to train the system—including JPEG artifacts already present in internet imagery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The research priorities behind the product
Rodriguez closes the prepared talk by connecting these questions to Krea’s research agenda. He says the company achieved its presented progress with eight people and had grown to twelve at the time of the talk.
The work he invites people to pursue includes aesthetics research, hyper-personalization and scaling real-time generative models across image, video, audio and 3D for users around the world. These are research and hiring priorities, not a deployment architecture demonstrated in the talk. They make the evaluation problem commercially concrete: a creative system must support many kinds of media while responding to individual tastes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning acceptable variation from examples
In the Q&A, Rodriguez restates the audience’s question as a search for perceptually aware metrics. Can a useful metric remain nearly unchanged when JPEG introduces artifacts that barely affect the viewer’s experience? He says such metrics have precedents in traditional encoding techniques, then turns from existing measures to something that could be learned.
The proposed approach is to train a classifier or continuous scoring model using examples of the intended judgment. Show it five images and label all five as good, even though they contain different artifacts. The acceptable variation need not be limited to JPEG compression. Those examples would give the evaluator evidence about which differences should not disqualify an image. This is a proposal for learning the criterion, not a reported result from a validated evaluator.
The ending returns to the opening hand from another direction. People can often recognize whether an image works before they can articulate a complete rule for it. Rodriguez’s wager is that this difficulty of specification makes the problem suitable for machine learning: examples can communicate a judgment that a checklist struggles to express. The task is to learn what people mean when they recognize a good image on sight.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Reproducible FID computation and examples showing how resizing and JPEG compression affect evaluation.
Shannon's foundational paper on communication, information, channels and noise.
Official code and examples for matching images with text and performing zero-shot image classification.
Further reading
- A New KreaArticle
Rodriguez and Perez describe Krea's creative-tool philosophy, company growth and research priorities.
Research on learned image similarity, with human judgments, LPIPS implementation links and a dataset.
Full transcript
Loading transcript…