The Tempting Pitch — and the Statistical Hole in It
Run your A/B test on a model instead of on users. Get results in hours, not weeks. Skip traffic allocation entirely.
Sounds great. But most proposals for replacing humans with LLMs in A/B tests quietly skip the question that makes experiments valid in the first place: under what conditions does the experiment actually identify the treatment effect you care about?
Randomized experiments are the gold standard because they causally identify the treatment effect by design. Swap real user responses for LLM-generated predictions, and that guarantee evaporates. Identification now holds only by assumption.
The framework for reasoning about this already exists — it's called surrogate endpoint theory, borrowed from biostatistics. In clinical trials, lab biomarkers serve as fast, cheap proxies for the real clinical outcome. The catch: a surrogate is only valid if it captures everything about the treatment that matters for the outcome.
For digital A/B tests, LLM predictions are the shiny new surrogate candidate. Whether they qualify is an empirical question, not a vibes question.
TL;DR: LLM predictions can stand in for human outcomes in A/B tests — but only by assumption, not by design. The conditions that make this work cannot be verified for new treatments, and they become less plausible the further the new treatment is from past experiments. The promise is least justified precisely when it offers the most benefit. For the full statistical treatment, see the original engineering write-up.
Raw LLM Predictions Are Biased — Not Just Noisy
The team empirically tested the promise using the Upworthy Research Archive, the largest open-access dataset of A/B tests currently available. It contains click-through rates for variants of news headlines across thousands of experiments.
They prompted gpt-4o-mini to predict the CTR of a typical user for each headline, separately for treatment and control variants. Then they plugged those raw predictions into a standard experimental analysis.
Result: they recovered only 39% of the observed human treatment effect.
If you treated those predictions as human data, you'd conclude treatments are less than half as effective as they really are.
This isn't random noise — the bias is systematic and directional. LLM outcomes attenuate treatment effects toward zero, making treatments look weaker than they are. Scale that across a product org and you get systematically wrong shipping decisions.
Two Conditions That Make LLM Outputs Valid Surrogates
The paper formalizes exactly two conditions under which LLM predictions can identify a human average treatment effect:
- Surrogacy. The LLM output must fully mediate the treatment effect on the human outcome. After conditioning on the LLM's prediction and any baseline covariates, treatment assignment tells you nothing additional about what the user would do. In plain English: the LLM captures everything about the treatment that matters for the human response. This is routinely assumed but rarely stated, and almost never validated.
- Comparability. The calibration function — the mapping from LLM predictions to human outcomes — must stay stable across the new experiment and the historical data used to fit it. If that mapping shifts when the treatment changes, calibration breaks.
When both hold, calibrating LLM outputs against real user A/B test data recovers the human treatment effect. When either fails, the estimate is biased — and not because of a lack of data. Even an infinite number of LLM predictions would only recover the effect on the LLM, not on users. Unlike random assignment, neither assumption is guaranteed by design.
Calibration Only Works With the Right Method
Not all calibration methods are equal. The team tested two on Upworthy data:
| Method | Result | Verdict |
|---|---|---|
| Linear calibration (OLS) | 3.8 standard errors from human benchmark | ❌ Too rigid — failed falsification test |
| Random Forest / Gradient-Boosted Trees | Within sampling error of human effect | ✅ Flexible enough to learn nonlinear mapping |
The takeaway: linear calibration can't capture the nonlinear relationship between LLM predictions and human behavior. Tree-based ML models can.
The Temperature Noise Problem
A single LLM prediction is noisy due to sampling temperature — the randomness in token generation. Left unaccounted for, this randomness biases the effect estimate toward zero and inflates its variance.
Drawing on measurement error theory, the fix is simple: sample multiple LLM outputs per experimental unit and average them. The noise component averages out, leaving a cleaner signal.
The Real Limitation: Future Interventions
Here's the uncomfortable part. Surrogacy and comparability can be partially assessed on historical data — but they can never be proven for a treatment you haven't tested yet.
This creates a brutal paradox:
- The further a new treatment departs from past experiments, the weaker the basis for trusting LLM output.
- For genuinely novel interventions — a new UI paradigm, a new pricing model, a feature unlike anything shipped before — the assumptions are inherently untestable.
- The setting where LLMs offer the most benefit is precisely where they're least likely to work.
Human experiments therefore remain indispensable for true product innovation.
Why Upworthy Is a Near-Ideal (and Misleading) Test Case
Upworthy is almost too favorable: binary outcome (click / no click), text-based treatments that are linguistically similar (headline variants), and LLMs trained on vast amounts of text about what makes headlines engaging.
For treatments that change layouts, algorithms, or pricing, the necessary conditions are much harder to justify. There is no empirical evidence that the conditions hold in general — across a company's portfolio of innovations — as would be required to use LLM-based A/B testing at scale.
Calibration Needs the Data You're Trying to Avoid Collecting
The framework doesn't eliminate user experiments. It only shows you can reduce how many you need — and only when new experiments resemble ones already run, letting extrapolation fill the gap. The upfront investment in real user responses isn't optional; it's what makes LLM-based A/B testing trustworthy at all.
Model Drift Breaks Calibration
One more wrinkle: any calibration function is fit to a specific model at a specific point in time. LLM providers update and replace models constantly. A calibration learned today may be invalid in six months — even for the same model. New calibration functions should ideally be fit on new user experiments, or they risk temporal bias.
Better LLMs, prompting, or fine-tuning don't escape this. Advances in LLM prediction are not a way out of human validation.
Final Word: By Design vs. By Assumption
User experiments work by design. LLM-based experiments work by assumption.
Like all proxy metrics, LLM surrogates work until the relationship between proxy and outcome shifts. The surrogacy framework makes that relationship explicit, testable, and clear about its consequences when it fails. It cannot guarantee the relationship holds for the experiment you care about most — the one testing something new.
That said, LLM predictions can make human experiments better:
- Filter weak ideas before they consume an experiment slot.
- Serve as covariates for variance reduction.
- Improve experiment selection and efficiency when you have strong historical data and the new treatment resembles past ones.
Substituting them for human outcomes is a different move entirely. It trades identification by design for identification by assumption. Don't give that up.
Where to Go Next
- Read up on surrogate endpoint theory in biostatistics — it's the cleanest mental model for proxy-metric validity.
- Study measurement error theory to understand why averaging multiple LLM samples matters.
- If you're running experiments at scale, look into variance reduction via covariate adjustment — it's the low-risk way to use LLM signal without giving up identification.