LLM Evals The Funnel That Makes Your A/B Tests Smarter (Not Redundant)

The New Superpower: Automated Quality Judgement at Scale

For years, A/B testing has been the gold standard for product decisions. At Spotify, only about 12% of A/B tests end in a shipped positive result—but 64% produce valid learning: a regression caught, an idea ruled out, a hypothesis refined. The win rate understates the value of experimentation.

Now we have a new capability: LLM evals. These automated judges can assess dimensions we couldn't scale before—relevance, coherence, tone, intent alignment—faster and cheaper than human annotation, on any data from test sets to A/B test variants.

The critical insight? Evals and experiments measure different things. The right relationship is a funnel, not a fork. As researchers Schultzberg and Ottens (2024) describe it, evals belong before your experiment, not instead of it.

Verification vs. Validation: Know the Difference

LLM evals verify: does the output conform to quality standards? Experiments validate: do real users respond as predicted?

A strong eval stack means you don't test to find out if the change does what you intend. Evals already told you that. You test to validate the intended change drives the business outcome it was meant to, and to bound the risk of harming the business.

# Simplified example: using an LLM judge to filter candidate prompts

def evaluate_candidate(prompt: str, judge_model: str = "gpt-4") -> dict:
    """
    Returns a quality score and flagged issues for a given prompt.
    This runs BEFORE the A/B test to discard low-quality variants.
    """
    response = openai.chat.completions.create(
        model=judge_model,
        messages=[
            {"role": "system", "content": "You are a quality judge. Rate the following prompt on relevance, coherence, and tone. Output JSON with score (0-1) and issues list."},
            {"role": "user", "content": prompt}
        ],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)

# Usage in a funnel
candidates = [prompt_a, prompt_b, prompt_c]
filtered = []
for c in candidates:
    result = evaluate_candidate(c)
    if result["score"] > 0.7:  # only pass high-quality candidates to A/B test
        filtered.append(c)
# Now run A/B test only on filtered candidates

Evals also generate hypotheses. Consider a team that builds an LLM judge to flag trust-breaking content—say a recommendation shared with a user it doesn't fit. The judge surfaces patterns the team didn't know to look for. Those patterns become product fixes. After the fix ships, the same judge can verify it worked: the flagged violations should drop. That's the eval doing two jobs: discovering what to improve, and confirming the improvement was realized.

Two Calibration Layers, One Feedback Loop

Evals are proxies. They substitute a score for an outcome you actually care about. That substitution is only valid as long as the score tracks the real outcome—the same dynamic we've described with proxy metrics.

Now LLM judges add a second calibration layer on top of traditional quantitative metrics (ranking scores, precision, recall). Both layers need validation against online outcomes. Both can drift. When the judge says Variant A is better, does it actually deliver a better user experience, or is the judge rewarding surface patterns that don't drive outcomes?

For example, when Anthropic released the Opus 4.5 model, Qodo's coding evals showed no improvement, but the model had improved substantially on longer tasks—a controlled experiment would have surfaced this. Miscalibration runs both ways. Without offline-online signal calibration, our evals are opinions, not evidence.

The Feedback Loop in Practice

By continuously adjusting the evals to improve their mapping to online outcomes, the evals become better and better verification tools. We are not ruling out that in the future, as AI develops, evals can map well enough to start acting as validations—but only if the offline/online calibration loop is in place.

Closing the Loop: From Opinions to Evidence

Teams under speed pressure sometimes call A/B tests "costly." We know from experience that shipping without an experiment can be incredibly costly, if a major regression in top business metrics goes undetected. The more complex the system, the more important it is to bound the risk.

Practical advice for your team:

1. Run evals early and often to find the best treatments.
2. Let the experiment validate that real users and systems respond as predicted.
3. Monitor metrics you didn't optimize for—guardrails catch regressions.
4. Run your LLM evals on the A/B test data itself. Did the version the judge preferred actually perform better with users? This extends the traditional evaluation funnel.

When the gap between eval scores and experiment outcomes is large, that's diagnostic gold. Each cycle helps calibrate the next. If users who received the improved version show better long-term engagement, the team has confirmed that what the judge measures actually matters. If the judge scores improved but user outcomes didn't, that's the calibration signal: the judge is capturing something, but not the thing that drives value.

Limitations & Caveats

• Evals struggle with long-running tasks and long-term behavior by construction. They are snapshots, not trajectories.
• LLM judges can hallucinate or exhibit bias. Always validate with human review on a subset.
• Calibration is not a one-time effort. As models and user behavior evolve, so must your evals.

Next Steps

• Start with a simple LLM judge for one dimension (e.g., tone consistency).
• Build a small pipeline that runs evals before every A/B test.
• After the test, compare eval scores with experiment outcomes.
• Iterate on your judge prompts and thresholds based on the gap.

For more on the architecture behind separating personalization and experimentation, check out our deep dive on why separate tech stacks for personalization and experimentation. And if you're curious how AI is moving from prototyping to production, see our analysis of Vercel's new v0 platform.

This article draws on internal research and practices at Spotify, originally published on Spotify Engineering.