Automate Custom PII Detection at Scale A Step-by-Step Guide with Amazon Macie and Step Functions

Why Automated PII Detection Is Non-Negotiable

In regulated industries like finance, healthcare, and government, data containing personally identifiable information (PII) flows through every application — from claims systems to partner feeds. Compliance with GDPR, HIPAA, CCPA, and PCI DSS requires you to know exactly where PII lives and how it's protected. Manual classification doesn't scale, and standard detection tools often miss domain-specific identifiers like policy numbers or medical record IDs.

Amazon Macie offers managed sensitive-data discovery, but you can extend it with custom identifiers tailored to your business. In this guide, you'll build an event-driven pipeline that scans every file the moment it lands in Amazon S3, using both built-in and custom identifiers, then generates audit-ready reports and alerts your team in real time.

What You'll Build

• EventBridge triggers on new S3 uploads
• Step Functions orchestrates the scan lifecycle
• Macie detects PII with built-in + custom identifiers
• Lambda handles compute logic between steps
• SNS sends notifications for high-severity findings

Architecture Overview

The solution uses a three-bucket pattern to isolate data by processing state: raw, staged, and scanned. This ensures unscanned data never mixes with validated data. Here's the flow:

1. A file lands in the raw bucket.
2. EventBridge detects the upload and starts a Step Functions workflow.
3. Step Functions copies the file to a staged bucket and creates a Macie classification job.
4. Macie scans using managed and custom data identifiers.
5. Lambda generates timestamped CSV and JSON reports in the scanned bucket.
6. SNS notifies your security team if high-severity PII is found.

Step-by-Step Implementation

Prerequisites

• AWS account with admin access
• AWS CLI v2 installed
• Amazon Macie enabled in your target region
• A valid email for SNS alerts

Step 1: Deploy the CloudFormation Stack

Download the sample CloudFormation template and deploy it via CLI or Console.

Option A: AWS CLI

# Deploy the stack with your desired parameters
aws cloudformation create-stack \
    --stack-name pii-pipeline \
    --template-body file://template.yaml \
    --parameters ParameterKey=CustomIdentifierPatterns,ParameterValue='{"PolicyNumber":"\\d{3}-\\d{2}-\\d{4}"}' \
    --capabilities CAPABILITY_IAM

Option B: Console

• Go to CloudFormation > Create Stack > With new resources
• Upload the template and fill in parameters
• Wait for CREATE_COMPLETE (3–5 minutes)

Step 2: Confirm SNS Subscription

You'll receive an email to confirm the subscription. Click Confirm subscription — otherwise alerts won't reach you.

Step 3: Verify Custom Data Identifiers

After deployment, check the Macie console to ensure your custom identifiers are active. For example, a custom regex for patient IDs or contract numbers will appear alongside managed identifiers.

Step 4: Test the Pipeline

Upload a sample file to the raw bucket:

# Upload a test file to the raw bucket
aws s3 cp sample-data.csv s3://pii-raw-bucket/

Watch the Step Functions execution start within seconds. The graph inspector shows each state: TriggerScan, WaitForMacie, CheckStatus, GetFindings, MoveFiles. Macie jobs may take 15–20 minutes depending on data size and identifier count.

Step 5: Review Reports

Once the workflow completes, you'll see two timestamped files in the scanned bucket:

• report-YYYYMMDD-HHMMSS.csv – summary of findings
• report-YYYYMMDD-HHMMSS.json – detailed findings

SNS sends an email notification if high-severity PII is detected, so your team can respond immediately.

Code Example: Lambda Function for Report Generation

The following Lambda function (Python) parses Macie findings and writes reports:

# Lambda function to parse Macie findings and generate CSV/JSON reports
import json
import boto3
from datetime import datetime

s3 = boto3.client('s3')

def lambda_handler(event, context):
    # Retrieve Macie findings from the event
    findings = event['findings']
    
    # Generate timestamped report data
    report_data = []
    for finding in findings:
        report_data.append({
            'finding_id': finding['id'],
            'type': finding['type'],
            'severity': finding['severity']['description'],
            'bucket': finding['resourcesAffected']['s3Bucket']['name'],
            'key': finding['resourcesAffected']['s3Object']['key'],
            'timestamp': datetime.utcnow().isoformat()
        })
    
    # Write JSON report to S3
    json_report = json.dumps(report_data, indent=2)
    s3.put_object(
        Bucket='pii-scanned-bucket',
        Key=f"reports/report-{datetime.utcnow().strftime('%Y%m%d-%H%M%S')}.json",
        Body=json_report
    )
    
    # Write CSV report to S3
    csv_lines = ['finding_id,type,severity,bucket,key']
    for item in report_data:
        csv_lines.append(f"{item['finding_id']},{item['type']},{item['severity']},{item['bucket']},{item['key']}")
    
    s3.put_object(
        Bucket='pii-scanned-bucket',
        Key=f"reports/report-{datetime.utcnow().strftime('%Y%m%d-%H%M%S')}.csv",
        Body='\n'.join(csv_lines)
    )
    
    return {'statusCode': 200, 'body': 'Reports generated'}

Limitations and Caveats

• Macie quotas: You can have up to 10,000 custom identifiers per account, but only 30 per classification job. If you need more, split them across multiple jobs.
• Job duration: Scanning can take 15–20 minutes or longer for large files. This isn't real-time in the strictest sense — it's near-real-time.
• Cost: Each classification job incurs Macie costs. For high-volume workloads, batch objects to reduce API calls.
• Security: This sample lacks production hardening. Always enable S3 default encryption, SNS encryption, and CloudTrail data events.

Next Steps

To extend this solution, consider:

• Data lake integration: Route JSON findings to Athena or QuickSight for trend analysis.
• Remediation workflows: Add a Step Functions branch to quarantine files when high-severity PII is found.
• Multi-account setups: Use cross-account EventBridge rules to centralize scanning.
• Batch processing: Modify the trigger to group objects into a single Macie job for cost optimization.

For more on AI-driven automation, check out our piece on Beyond the Chatbot: How Cloudflare's Agent Lee Redefines Platform Interaction — and if you're working with AI agents, see ADK Go 1.0: Production-Grade AI Agents with Observability.

Conclusion

Automating PII detection with Amazon Macie and Step Functions isn't just about compliance — it's about building trust. This pipeline gives you real-time visibility into sensitive data, reduces manual effort, and ensures you're ready for audits. Deploy it, customize your identifiers, and extend it to fit your organization's needs.

References

• AWS Architecture Blog – Original Article