Your ML Pipeline Scored 0.98 in Cross-Validation. It Was Lying.

The Pipeline We All Write

If you've used scikit-learn for a while, you've probably written this pipeline: numeric columns get imputed and scaled, everything else gets imputed and one-hot encoded, and columns are routed by dtype. It works on almost any table without you looking at it.

It's short, reusable, and easy to trust. That's the problem.

I wanted to find out what this pipeline silently gets wrong, so I tried to break it on purpose. The result: a model that scored 0.979 in 5-fold cross-validation and 0.564 on new customers. Nothing crashed. No warning appeared. Every number looked like a strong model.

This is the story of four traps a dtype-based pipeline cannot see, and the self-checking automation I built to catch them. For a deeper look at why runtime guardrails alone aren't enough for AI systems, see our analysis of the missing trust layer for AI agents.

Four Traps, One Silent Failure

I generated synthetic customer-churn data (8,000 rows per dataset, 5 seeds) and planted four traps:

1. Region as a number. region_code holds integers 1–12, but each region affects churn differently and not in order. The pipeline scales it like a measurement.
2. Informative missing values. support_calls is missing for ~45% of churners and 8% of non-churners. Median imputation erases that signal.
3. A date stored as text. signup_date is a YYYY-MM-DD string, so it becomes a one-hot column per unique date (1,931 unique values in 6,000 rows).
4. An ID that leaks the label. In training, customer_id was assigned roughly in order of outcome. In the clean holdout, IDs are unrelated to churn.

The Score That Lied

Inside the logistic regression for seed 0, the single strongest feature out of 1,940 was customer_id (|coefficient| 5.05). The next strongest were individual signup-date one-hot columns. The model read the label off the ID and memorized per-date noise.

Teaching the Pipeline to Check Itself

If you build automation for people who don't code, "remember to check for leaks" is not a plan. Every trap above leaves a statistical fingerprint. Here's a scanner that runs before modeling:

import pandas as pd
import numpy as np

def scan_for_traps(df, target_col, id_like=('_id', 'id_'), date_like=('date', '_at', 'time')):
    """Print plain-English warnings for common leakage traps."""
    warnings = []
    y = df[target_col]

    for col in df.columns:
        if col == target_col:
            continue
        s = df[col]

        # Trap 4: ID-like column correlated with target
        if any(tok in col.lower() for tok in id_like) and s.dtype.kind in 'iu':
            corr = np.corrcoef(s.fillna(-1), y)[0, 1]
            if abs(corr) > 0.3:
                warnings.append(f"'{col}' looks like an ID but correlates with target (r={corr:.2f}). Possible leak.")

        # Trap 3: text column with too many unique values (likely a date)
        if s.dtype == object and s.nunique() > 0.1 * len(s):
            warnings.append(f"'{col}' is text with {s.nunique()} unique values. Parse it before one-hot encoding.")

        # Trap 2: informative missingness
        if s.isna().any():
            miss_rate = y[s.isna()].mean()
            present_rate = y[s.notna()].mean()
            if abs(miss_rate - present_rate) > 0.1:
                warnings.append(f"'{col}' missingness is informative (target rate {miss_rate:.2f} vs {present_rate:.2f}). Add a missing indicator.")

        # Trap 1: numeric column with low cardinality (likely a category)
        if s.dtype.kind in 'iu' and s.nunique() < 20 and col.lower().endswith(('code', 'id', 'zip')):
            warnings.append(f"'{col}' is numeric but looks categorical ({s.nunique()} values). Treat as category.")

    for w in warnings:
        print('⚠️', w)
    return warnings

def gap_warning(cv_auc, holdout_auc, threshold=0.1):
    """Flag suspicious CV/holdout gaps."""
    gap = cv_auc - holdout_auc
    if gap > threshold:
        print(f"⚠️ Cross-validation ({cv_auc:.2f}) is much higher than holdout ({holdout_auc:.2f}). Suspect leakage or memorization.")
    return gap

On seed 0, this flagged exactly the four planted columns (customer_id, region_code, support_calls, signup_date) and left tenure_months, monthly_spend, and plan_type alone.

Where This Approach Falls Short

• One synthetic data source. All results come from one generator I wrote. The direction of effects should carry over, but sizes will differ on real data. Fix: run the same variants on a real table you know well.
• The ID leak is planted and exaggerated. Real leaks are usually weaker. Fix: check every ID-like or timestamp-like column for its relationship with the target before modeling.
• Only two model families. Other models may react differently. Fix: repeat with your own model, using a holdout built the way your future data will arrive.
• Spread in some results. Gradient boosting's missing-indicator and region numbers are noisy. Fix: use more seeds or repeated CV before drawing conclusions about a single change.
• The checker was tested on the same synthetic data. Real data will trigger warnings that need a human decision. Fix: treat warnings as prompts to look, not as automatic fixes.

Five Questions Before You Trust Automation

1. Is there a holdout that looks like future data? Cross-validation on training data cannot catch a leak that exists in the training data.
2. Which column does the model lean on most? If an ID, row number, or timestamp is on top, stop.
3. Is a numeric column really a category? Codes, zip codes, and region IDs are numbers that are not quantities.
4. Are the missing values random? If the target rate differs where a column is missing, add a missing indicator.
5. Are there text columns with thousands of unique values? Dates, names, and free text need their own handling, not one-hot encoding.

Automation Isn't Verification

An 8-line pipeline is a good starting point and a poor stopping point. In this experiment it produced a 0.98 that fell to 0.56 on new data, and it took four separate fixes — not one — to reach an honest 0.83.

Automation saved time, but it did not do the checking, so I added the checking to the automation. If you build tools for people who can't spot these problems themselves, that is the part to build in.

Next steps: wrap the scanner above into a Pipeline preprocessor step, log warnings to your experiment tracker, and gate model promotion on a CV/holdout gap check. For a related pattern in production systems, see how CSS @function brings native custom functions without Sass — a reminder that convenience without verification is a debt.

Source: Towards Data Science — My Machine Learning Pipeline Cheated, and Cross-Validation Never Noticed