How Netflix Built a Real-Time Service Map From Silos to Living Topology

The 3 AM Problem Every Distributed Systems Engineer Knows

Your phone buzzes. A critical service is showing elevated error rates. You need to know: Is this my fault? What depends on me? Where do I even start looking?

Netflix engineers faced this exact scenario repeatedly. With thousands of microservices powering everything from personalized recommendations to live sports streaming, the traditional observability stack — metrics, logs, traces — showed fragments of the picture. But no single tool answered the fundamental question: how does everything connect?

This is the story of how Netflix built Service Topology, a living map of their distributed infrastructure that updates in real-time as services deploy, traffic patterns shift, and new dependencies form.

Why Traditional Observability Falls Short

Most observability tools are great at showing symptoms but terrible at showing structure:

• Metrics tell you a service is failing, but not who it depends on.
• Logs show individual service behavior, but not the call graph.
• Traces follow single requests, but don't give you the steady-state topology.

Netflix analyzed thousands of support requests over four years. The pattern was clear: engineers constantly asked dependency questions — upstream/downstream relationships, blast radius, root cause propagation. They needed a unified view, not a mental jigsaw puzzle.

The Three Sources of Truth

The key insight? No single data source tells the complete story. Netflix built Service Topology by combining three complementary layers, each compensating for the others' limitations:

1. eBPF Network Flows (Network Layer)

• What it captures: Kernel-level network flow records — every connection between services, regardless of instrumentation.
• Strength: Comprehensive coverage. Every service shows up because you're capturing actual traffic.
• Weakness: No application context. You know Service A connected to Service B, but not which API endpoint was called.

2. IPC Metrics (Application Layer)

• What it captures: Inter-Process Communication metrics from instrumented services — gRPC, GraphQL, REST calls.
• Strength: Rich context — specific endpoints, error rates, latency, protocol details.
• Weakness: Only works for instrumented services. Non-instrumented code is invisible.

3. End-to-End Tracing (Request Layer)

• What it captures: Distributed traces aggregated to show actual request paths through the system.
• Strength: Shows runtime behavior — conditional logic, feature flags, real request flows.
• Weakness: Sampling. Rarely-used code paths may be missed.

By querying all three layers in parallel and merging results, engineers get a unified graph that is both complete (network flows) and context-rich (IPC + tracing).

Architecture: From Raw Flows to a Queryable Graph

flowchart LR
    A[Kafka Flow Logs] --> B[Pekko Streams Processing]
    B --> C[Stage 1: Initial Aggregation]
    C --> D[Stage 2: Intermediary Resolution]
    D --> E[Stage 3: Final Aggregation]
    E --> F[Graph Database]
    F --> G[gRPC API]
    G --> H[Unified Topology View]
    H --> I[Engineers & Automated Systems]

Here's the high-level pipeline Netflix implemented:

1. Multi-Region Ingestion: Flow logs from Kafka across AWS regions are consumed continuously.
2. Distributed Processing: Apache Pekko Streams (Akka fork) handles fault-tolerant, backpressure-aware processing.
3. Three-Stage Aggregation: The critical challenge — network flow logs show individual hops (App A → Load Balancer → App B), not direct application connections. The aggregation pipeline:
   • Stage 1: Initial aggregation from Kafka.
   • Stage 2: Identifies network intermediaries (load balancers, NAT gateways, proxies) and reconstructs direct paths.
   • Stage 3: Final aggregation with health status integration before persistence.
4. Graph Storage: Netflix's custom graph database, built on distributed key-value storage, supports fast multi-hop traversal. Each data source creates a physically separate graph partition, allowing parallel queries.
5. gRPC API: Exposes topology with filtering (availability tier, business domain), pagination, and sub-second response times.

Key Engineering Decisions

• Physical separation of graph layers allows independent evolution and parallel querying.
• Time-window aggregation enables time travel — querying historical topology without exploding storage costs.
• Hot node prevention: The three-stage approach distributes load across multiple points, even when specific services see 100x more traffic.

What Engineers Can Do Now

Service Topology is already in production at Netflix, helping engineers:

• Visualize dependencies with filters for availability tier and business domain.
• Jump to detailed signals (logs, traces, metrics) directly from the topology view.
• Understand blast radius before taking a service down for maintenance.
• Overlay health status to identify cascading failures.
• Query programmatically via gRPC API for automated resilience frameworks.
• Time travel to understand what changed around the time an issue started.

Limitations and Caveats

No system is perfect. Here are the trade-offs Netflix acknowledges:

• eBPF network flows lack application context — you see that a connection happened, but not what was called.
• IPC metrics only cover instrumented services — non-instrumented code creates blind spots.
• Tracing sampling may miss rarely-used code paths in aggregated views.
• Graph storage at scale requires careful partitioning and query optimization to maintain sub-second response times.

Next Steps: Automated Root Cause Analysis

Netflix is already working toward the next frontier: automated root cause analysis. By combining the topology graph with change event overlays (deployments, config changes), they envision an intelligent agent that continuously crawls dependencies, correlates failures, and surfaces likely root causes automatically.

What This Means for Your Architecture

Whether you run 10 microservices or 10,000, the core insight applies: observability is not just about signals — it's about structure. Before you add more dashboards, ask yourself: Do my engineers have a living map of their dependencies?

For a deeper dive into the engineering challenges (Kafka lag, GC pauses, reactive stream debugging), check out the original Netflix Tech Blog post.

Further Reading

• Gemini for Home: Proactive AI Service Providers — How AI is transforming service discovery and automation.
• Evaluating LLM-Generated Customer Journeys with CDP Metrics — Measuring AI-generated workflows with structural data.

Key Takeaways

1. Combine multiple data sources — no single source (eBPF, IPC, tracing) gives a complete picture.
2. Real-time matters — static diagrams are archaeology, not observability.
3. Physical separation of graph layers enables parallel querying and independent evolution.
4. Time-travel capability is essential for debugging — understanding what changed is as important as knowing the current state.
5. Programmatic access enables automated blast radius calculation and incident response.

Learning Path

If you want to build something similar for your own systems:

1. Start with distributed tracing (OpenTelemetry) to understand request flows.
2. Add eBPF-based network monitoring (Pixie, Cilium) for coverage of uninstrumented services.
3. Build a graph data model that can merge multiple sources.
4. Implement time-window aggregation for historical queries.
5. Expose a unified API that both humans and automation can consume.