Scaling Headless Browsers at Cloudflare From KV Bottlenecks to D1 + Queues

Why This Migration Matters

Browser Run lets developers programmatically control headless browsers on Cloudflare’s global network. It's used for end-to-end testing, URL investigation, PDF rendering, and—increasingly—as the backbone for AI agents that need to interact with the web. As demand exploded (especially from agent builders), the old shared infrastructure with Browser Isolation (BISO) became a bottleneck.

Key pain points before the migration:

• Slower startup times due to larger BISO container images.
• Poor global distribution leading to latency spikes.
• Scaling conflicts between BISO's long sessions and Browser Run's short, spiky usage.

The solution? Build on Cloudflare's own platform: Durable Object-enabled Containers.

Reference: Cloudflare Browser Run official blog

The Migration: Containers + DO

The team started by inserting a Worker in the request path to route a subset of users to Container-powered browsers while keeping BISO as a fallback. This dual-mode approach allowed them to compare performance and isolate bugs.

Key Architectural Challenge

DO-enabled Containers create a Durable Object close to the request, but the actual container can spin up far away. For screenshot requests that exchange dozens of WebSocket messages, those extra milliseconds add up.

Solution: Regional pre-warmed pools of DO-backed browser containers. When a request arrives, the system picks the DO-container pair closest to the user within that region.

// Simplified: pick the best container based on region and latency
async function pickContainer(userLocation) {
  const region = getNearestRegion(userLocation);
  const pool = await getWarmPool(region);
  // Use D1 to atomically claim a container
  const container = await claimContainer(pool, region);
  return container;
}

From KV to D1 + Queues

Initially, container state was stored in KV. But KV's eventual consistency (~30s cache TTL) caused race conditions: a container marked "available" might already be claimed by the time the request reached it.

The fix: Migrate to D1 with transactional SQLite semantics. Here's the core acquisition query:

WITH candidate_pool AS (
  -- pick based on latency and other rules
)
UPDATE containers
SET status = 'picked'
WHERE sessionId IN (
  SELECT sessionId
  FROM candidate_pool
  ORDER BY RANDOM()
  LIMIT ?5
)
RETURNING data;

Batch Writes for Scale

Each container updates its state every 5 seconds. At 1ms per write, a single D1 instance can only handle ~5,000 containers. By batching writes via Queues (100 rows per batch), they achieved 0.1ms P95 batch writes, supporting up to 500,000 containers per location.

{
  "queues": {
    "consumers": [
      {
        "queue": "production-core-containers-queue-weur",
        "max_batch_size": 100,
        "max_batch_timeout": 1,
        "max_retries": 1
      }
    ]
  }
}

This architecture gives them 4x more concurrent browsers (up to 120) and 50% faster Quick Action response times.

For more on how to protect users from malicious links, check out How Advanced Browsing Protection Checks URLs Without Compromising Privacy.

Additional Perks & Results

Optimized Quick Actions

With dedicated containers, the team could send all parameters in a single HTTP request instead of a multi-step WebSocket dance. This cut average response times sharply.

Faster Browser Upgrades

No longer tied to BISO's upgrade schedule, they now ship Chrome updates and new features (like WebGL and WebMCP) on their own cadence.

Limitations & Risks

• Queue backlogs can still cause stale state. Fallback regions mitigate this, but it adds complexity.
• D1 sharding per location requires careful capacity planning. The team notes that batch writes are essential to avoid overloading the database.
• DO-Container distance remains a challenge for latency-sensitive operations, though regional pools help.

Next Steps for Learners

• Dive into Cloudflare Workers Docs to understand DO and D1.
• Experiment with Browser Run Quick Actions to see the performance improvements firsthand.
• Explore the Agents SDK if you're building AI agents that need browser interaction.

Conclusion

Cloudflare's Browser Run team turned a scaling crisis into a platform success story. By migrating from shared BISO infrastructure to dedicated Containers with Durable Objects, and solving real-time state management with D1 + Queues, they achieved dramatic performance gains. The key lesson: when building at scale, own your infrastructure and batch your writes.

If you're interested in autonomous AI agents, also check out Meta's REA: The Autonomous AI Agent That's 5x More Productive at ML Experimentation.