Serverless Genomics at Scale How CSIRO Queries Billions of Variants for $0.40/Month

Why Genomic Data Sharing Is a Cloud Architecture Problem

Genomic data is exploding. Clinical diagnostics, mega-biobanks, and rare-disease research all demand that we query hundreds of millions of individuals across billions of genomic locations — while keeping sensitive data under the control of its original custodians.

The GA4GH Beacon protocol was designed exactly for this: a federated API standard for discovering genomic and phenotypic data without centralizing it. But a standard is only as good as its reference implementation.

CSIRO, Australia's national science agency, built sBeacon — a production-ready, fully serverless Beacon implementation on AWS. It scales to mega-biobank cohorts, ingests standard VCF files without transformation, and answers real-world queries in about 1.5 seconds.

The cost? Roughly USD 0.40 per month for a 1000 Genomes-scale dataset. Let's break down how that's actually possible.

The Architecture: Two Flows, Zero Servers

sBeacon splits into two clean processes: data onboarding and data querying. Everything runs on Amazon S3, AWS Lambda, Amazon DynamoDB, and Amazon Athena.

Data Onboarding — Index, Don't Copy

The key insight: genomic data never leaves its original S3 bucket. Only metadata (disease status, age, location) is indexed.

# Simplified onboarding flow (pseudo-code mirroring sBeacon's Lambda orchestration)
# 1. User submits dataset location to API Gateway
payload = {
    "vcfLocations": ["s3://custodian-bucket/cohort.vcf.gz"],
    "metadataSource": "ontoserver-endpoint"
}

# 2. Lambda writes metadata to S3 in ORC format
write_orc(metadata, "s3://sbeacon-bucket/metadata/")

# 3. Ontology index built via Ontoserver (or Ensembl OLS V4 compatible)
build_ontology_index(terms)

# 4. Athena CTAS builds queryable metadata tables
athena.execute("""
    CREATE TABLE AS SELECT *
    FROM metadata_orc
    WHERE ...
""")

# 5. VCF files are referenced, not copied — tabix index enables random access

Because only metadata is duplicated, ingestion cost is dominated by a few megabytes of ORC files, not gigabytes of sequence data.

Data Querying — Fan-Out with Lambda

When a query arrives, API Gateway routes it to a Microservice Lambda. That Lambda resolves ontology terms in DynamoDB, queries Athena for metadata, and — if variant data is needed — fans out across VCF files:

# Variant Querying Module — fan-out pattern
def initiator_handler(event, context):
    # Fan out splitQuery across VCF files
    for vcf in event["vcfLocations"]:
        invoke_lambda("splitQuery", {"vcf": vcf, "query": event["query"]})

    # Each VCF spawns performQuery across genomic regions
    # performQuery uses bcftools + htslib HTTP byte-range reads
    # Only ~1 KB is fetched per query — the raw VCF never enters Lambda memory

The magic is in htslib's HTTP byte-range requests: performQuery passes the S3 URL directly to bcftools, which fetches only the tabix-indexed region of interest. The raw genomic sequence bytes never flow through Lambda as returnable data.

The Cost Math (chr1, 2504 genomes)

Query time stays flat as result count grows — from 4 variants to 400 variants, response stays around 1.5 seconds.

Zero Trust, Federated by Design — and Where It Breaks

Security Model

sBeacon enforces a zero trust model where no component, user, or request is implicitly trusted:

• Explicit auth: Every request carries a JWT from Amazon Cognito, validated at API Gateway before any Lambda runs.
• RBAC via Cognito groups mapped to disclosure tiers:

• Ephemeral compute: Lambda's /tmp is cleared between cold starts, bcftools subprocess runs within a 10s timeout, and no state persists.
• Decentralized storage: A dataset's vcfLocations are S3 URIs that can point to buckets owned by entirely different organizations. Each institution deploys its own Terraform stack — no shared infrastructure, no central data lake.

⚠️ Limitations You Must Plan For

1. Whole-genome range queries time out. API Gateway has a hard timeout. If someone accidentally tries to scan the entire genome, the request fails. Functional operations across a full genome require a different architecture (e.g., Step Functions for long-running orchestration).

2. Lambda concurrency can be exhausted. A single fan-out query spawns many parallel invocations. Burst queries can starve other functions in the same AWS account. Monitor the ConcurrentExecutions metric at both account and function level.

3. Silent throttle loss. Synchronous Lambda invokes don't auto-retry on throttle. A 429 from performQuery means the result is silently lost unless you handle it. Set CloudWatch alarms on the Throttles metric — or wire up an error-catcher pattern that emails diagnostics on failure.

4. Cold starts after idle periods. Burst fan-out after idle time adds latency. Provisioned concurrency helps, but raises idle cost.

Why Lambda Instead of Step Functions?

The team explicitly chose Lambda over Step Functions because Lambda handles much larger payloads — critical given the size and complexity of genomic data and the fan-in/fan-out pattern. For this workload, Lambda was both cheaper and more flexible.

The Takeaway: Federated Data Platforms Are a Solved Pattern

sBeacon is a rare case study where the architecture choices are directly traceable to cost and latency numbers:

• Don't copy what you can index. Referencing VCFs in place instead of ingesting them dropped storage to pennies.
• Byte-range reads beat data lakes. htslib's HTTP Range requests fetch ~1 KB per query — no ETL, no centralization.
• Serverless fan-out fits genomics. Parallel Lambda invocations map naturally to the "scan many files, return small results" shape of variant queries.
• Zero trust isn't a buzzword here. Cognito groups, JWT claims, and per-institution Terraform stacks enforce it structurally.

If you're building genomics on AWS, you can deploy sBeacon against existing S3-hosted VCF data and progressively federate with other Beacons for cross-institutional discovery.

Next steps to explore:

• Read the sBeacon paper ("Scalable genomic data exchange and analytics with sBeacon") for benchmark methodology.
• Clone the terraform-aws-serverless-beacon repository and run a local deployment against a small VCF.
• Study the error-catcher pattern for surfacing Lambda failures in fan-out architectures.

For a different angle on how capacity planning and observability failures play out in production pipelines, see our breakdown of a real-world media pipeline outage. And if you're tracking how AI is reshaping service platforms, Gemini for Home shows how proactive AI is becoming a full-stack concern.

Source reference: AWS Architecture Blog — How CSIRO built scalable, cost-optimized genomic variant querying on AWS