How Meta Migrated Its Petabyte-Scale Data Ingestion System Without Data Loss

The Challenge: Migrating a Hyperscale Data Ingestion System

Meta's social graph relies on one of the largest MySQL deployments in the world. Every day, their data ingestion system incrementally scrapes several petabytes of social graph data into the data warehouse to power analytics, reporting, and machine learning model training. As the company grew, the legacy system—built on customer-owned pipelines—became unstable under increasingly strict data landing time requirements.

The team needed to migrate tens of thousands of ingestion jobs to a new, simpler self-managed data warehouse service that could operate efficiently at hyperscale. The migration had to be seamless: no data loss, no latency regression, and no resource usage spikes.

Key constraints:

• Zero tolerance for data quality issues (row count & checksum must match)
• No landing latency regression allowed
• Compute and storage usage must not increase
• Rollback must be fast and safe

This is not a theoretical exercise—it’s a battle-tested playbook from one of the most demanding production environments on the planet.

The Solution: A Three-Phase Migration Lifecycle

To ensure data integrity and operational reliability, Meta established a clear migration lifecycle with three phases:

Phase 1: The Shadow Phase

Each job was first run in a pre-production environment as a "shadow job." The shadow job consumed the same source data as the production job but wrote to a separate shadow table. This exposed the new system to real production behavior while providing an isolated space to inspect outcomes and deploy fixes.

Critical monitoring: Row count and checksum were continuously compared between the production job and the shadow job. Any mismatch triggered an immediate investigation.

# Simplified example: comparing row count and checksum between production and shadow tables

def validate_data_quality(prod_table, shadow_table):
    """
    Compare row count and checksum between production and shadow tables.
    Returns True if both match, otherwise logs mismatches for debugging.
    """
    prod_row_count = get_row_count(prod_table)
    shadow_row_count = get_row_count(shadow_table)
    
    prod_checksum = get_checksum(prod_table)
    shadow_checksum = get_checksum(shadow_table)
    
    if prod_row_count != shadow_row_count:
        log_mismatch("Row count", prod_table, shadow_table, prod_row_count, shadow_row_count)
        return False
    
    if prod_checksum != shadow_checksum:
        log_mismatch("Checksum", prod_table, shadow_table, prod_checksum, shadow_checksum)
        return False
    
    return True

Phase 2: The Reverse Shadow Phase

Once the shadow job ran reliably, the roles were swapped: the shadow job’s data was written to the production table, and the original production job’s data was written to the shadow table. This gave two key benefits:

• Ongoing data-quality signals by continuing to compare outputs
• Instant rollback if discrepancies were detected, without needing to recreate the old job

Phase 3: Migration Cleanup

After both jobs ran in the reverse shadow configuration without issues, the old system’s shadow job was removed. The new system took over completely.

Custom Data Quality Analysis Tooling

Meta built a comprehensive debugging toolkit. For each landed shadow table partition, the system read the corresponding production partition and compared row count and checksum. Mismatches were logged to Scuba (Meta’s real-time data management system). Every hour, the tool read the logs, ran queries to identify example rows causing mismatches, and logged detailed debugging info back to Scuba.

This same tool is still in use after the migration as part of the release validation process.

Handling Rollout and Rollback

Because both systems used change data capture (CDC), problematic data could propagate to newly generated data. To reduce risk, Meta focused on:

1. Early signals before problematic data lands on consumers
2. Stopping the bleeding quickly during rollback

Early signals: After rollout, the system triggered backfill on both production and shadow jobs. If results matched, the migration was successful. If not, the job was rolled back immediately—before data consumers were impacted.

Stopping bad data propagation: During the reverse shadow phase, if a specific partition had data quality issues, it was marked in metadata. For delta partitions, new data would stop landing and an alert was sent. For target partitions, the system selected an older partition and merged it with more deltas. Rollback could then query the metadata to find all bad partitions and fix them with backfill.

Executing at Scale: Automation and Batching

With tens of thousands of ingestion jobs to migrate, manual migration was impossible. Meta built:

• External migration tools that continuously monitored job status signals and automatically promoted/demoted jobs between lifecycle stages
• System-level and job-level dashboards for tracking progress and debugging

Because migration capacity was limited, jobs were migrated in batches. Jobs were categorized by throughput, priority, and special cases. Teams avoided creating shadow jobs with known issues to prevent unnecessary full dumps (which are slow and expensive). They also reused snapshot partitions delivered by the old system as initial snapshots to reduce the full dump load.

For a deeper dive into progressive rollout strategies for production changes, check out our guide on Progressive Rollouts in Vercel Flags.

Limitations and Caveats

While Meta’s approach is robust, it’s important to understand its constraints:

• High operational overhead: Running dual systems (shadow + production) requires significant compute and storage resources. Not every organization can afford to double their infrastructure during migration.
• CDC-specific design: The rollback strategies rely heavily on CDC metadata. If your system doesn’t use CDC, you’ll need to adapt the approach.
• Complex tooling: Building the custom data quality analysis tool and automated migration orchestrator is non-trivial. Smaller teams may need to leverage existing open-source solutions like Apache Airflow or dbt for similar validation.
• Batch dependency: The batching strategy assumes you can prioritize jobs. If all jobs have equal priority, you may face contention for limited shadow capacity.

Next Steps for Learning

If you’re planning a similar migration, here’s a recommended learning path:

1. Understand your CDC pipeline: Read about change data capture patterns in your data warehouse (e.g., Snowflake Streams, BigQuery Change Tracking).
2. Build a shadow testing framework: Start with a single job and validate row counts and checksums before scaling.
3. Automate the lifecycle: Implement a state machine for migration stages (shadow → reverse shadow → cleanup) with automated promotion criteria.
4. Practice rollback simulations: Test your rollback procedure in a staging environment before going to production.

For a related case study on managing complexity in distributed systems, see Deconstructing Complexity: A Multi-Agent Architecture for Intelligent Advertising.

Conclusion

Meta’s data ingestion system migration is a masterclass in large-scale system migration. The key takeaways are:

• Establish a clear migration lifecycle with defined success criteria at each stage
• Use shadow testing to validate correctness without impacting production
• Implement reverse shadow for ongoing quality signals and instant rollback
• Automate everything—monitoring, promotion, and rollback—when dealing with tens of thousands of jobs
• Batch strategically to avoid wasting resources on jobs with known issues

This playbook isn’t just for Meta-scale systems. Any team migrating a critical data pipeline can adopt these principles, adapted to their scale and tooling. The underlying philosophy—validate early, rollback fast, automate relentlessly—is universal.

Source: Meta Engineering Blog - Migrating Data Ingestion Systems at Meta Scale