Agent Platform Evaluations Hit GA A Deep Dive Into Adaptive Rubrics and Online Monitors

Why Agent Evaluation Is Harder Than Model Evaluation

You can benchmark a model on a static test set and call it a day. You cannot do that with an agent. An agent's quality depends on the tools it calls, the multi-turn flow it produces, and the inputs real users throw at it — none of which exist at training time.

The Agent Platform team at Google Cloud just shipped Agent and Model Evaluations to GA, and the framing is refreshingly honest: an agent that's green across your whole test suite can still drift a week after launch. That's not a testing failure — it's the reality of production inputs being bigger and stranger than anything you wrote a case for.

This article walks through what actually landed, how the pieces fit together, and where the sharp edges are. For a comparable deep dive on how large-scale systems get migrated without breaking, see our write-up on migrating petabyte-scale data ingestion systems at Meta.

The Core Primitive: Experiments, Metrics, Traces

The unit of work is the experiment: a dataset of eval cases plus a set of metrics and scores you run and review as you iterate.

Three things make it usable in practice:

1. Same engine, offline and online. Metrics you define locally run unchanged in production monitors. That's the whole point — if scores shift in prod, it's the agent, not the measurement.
2. Transparent artifacts. Every experiment artifact lands in Cloud Storage, so you can version and audit runs for compliance.
3. Reachability from wherever you build. SDK, agents-cli, the Cloud console, ADK, and even a reusable skill for Claude Code-style agents.

Bringing Your Own Metric

Beyond the 20+ pre-built metrics (ROUGE, BLEU, MetricX, COMET, exact match), you can define your own:

• Code-based metric — a plain Python function. Exact text match, JSON-shape check, whatever you can express.
• LLM-as-a-judge metric — your criteria, your rating scale, your judge model. Locally you can use any provider; server-side supports any Model Garden model including Gemini and Anthropic.

# Example: a code-based metric that checks JSON shape compliance
import json

def json_shape_metric(response: str, expected_keys: list[str]) -> float:
    """Return 1.0 if the response is valid JSON containing all expected keys."""
    try:
        data = json.loads(response)
    except json.JSONDecodeError:
        return 0.0
    missing = [k for k in expected_keys if k not in data]
    return 1.0 if not missing else 0.0

Adaptive Rubrics (The Interesting Part)

Co-developed with Google DeepMind researchers, adaptive rubrics generate case-specific pass/fail tests from three inputs:

• the eval case definition,
• the developer instruction,
• the tool declarations.

It then grades traces against those rubrics and returns a verdict plus a rationale per rubric. You can steer generation with natural-language guidelines, and the resulting rubric group is reviewable and reusable across agents.

Out of the box you get calibrated variants for the things you usually want to know: Task Success, Tool Use Quality, Safety (hate speech, harassment, dangerous content, explicit material, PII), Trajectory Quality, Final Response Quality, Hallucination, and Grounding.

What's Actually Missing (And Where It Bites)

A few honest caveats before you rip out your bespoke eval harness:

Two things that will surprise teams migrating from a homegrown setup:

1. Issue clustering is the sleeper feature. Instead of staring at 4,000 failed evals, the system groups them into interpretable clusters against your taxonomy — or a pre-built adaptive-rubric taxonomy if you don't have one yet. That's the difference between "we have evals" and "we act on evals."
2. The environment simulator is the missing piece for agent testing. Point it at a tool, tell it to return mocked data, a forced error, or added latency, and it intercepts the call during the run. You finally get to test how the agent behaves when the backend is slow or broken — without touching production.

For a parallel case study in designing systems that fail safely under load, our piece on how Messenger's advanced browsing protection works without compromising privacy is worth reading alongside this one.

The Takeaway

Evaluations in Agent Platform are now GA, and the design choices reveal a clear thesis:

• Consistency beats sophistication. One engine for local experiments and live traffic means a drift in production points at the agent, not the measurement.
• Bring your own everything. Code metrics, LLM-judge metrics, judge models from any provider, taxonomies — all land in the same versioned org-wide registry.
• Generate, don't hand-write. Case generators, user simulators, and environment simulators exist because hand-written test suites cap your coverage at what you thought to type.

Next Steps

1. If you're on ADK, wire eval sets into pytest and run them in CI today — it's the lowest-friction starting point.
2. If you already collect traces via Cloud Trace, click the eval button in the traces UI on a real session and see what the default metrics say.
3. Set up an online monitor with sampling on 5–10% of traffic and drift alerts to Slack before you scale it up.

Source: Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA