AI Data Services: How a Fintech Improved Model Accuracy by 23% With Clean Training Data

Context: A High-Stakes Fraud Problem Hitting a Performance Ceiling

A mid-sized fintech—here called Northbridge Payments—processed a mix of card-not-present transactions, instant transfers, and account-to-account payments for merchants operating across multiple regions. Fraud losses were manageable, but operational costs were rising fast: more false positives meant more manual reviews, more customer friction, and more chargeback disputes.

Northbridge had invested heavily in machine learning to detect suspicious activity in near real time. The fraud detection model sat behind multiple product touchpoints, flagging transactions for:

• Automatic decline
• Step-up authentication
• Manual review
• Allow with monitoring

Over several quarters, Northbridge’s data science team made architectural improvements—feature engineering, hyperparameter tuning, and model selection iterations. Yet performance plateaued. The model still underperformed in production: fraud slipped through, while legitimate users faced avoidable friction.

The team faced an uncomfortable possibility: the model wasn’t the main problem.

Challenge: Strong Modeling Effort, Weak Real-World Outcomes

Internally, model metrics looked “acceptable,” but outcomes were inconsistent by segment:

• New merchants behaved differently from established ones.
• Cross-border transfers showed a distinct fraud pattern.
• Certain transaction types were over-flagged despite low fraud rates.

Two operational symptoms raised red flags:

1. Model retrains didn’t reliably improve production outcomes
   Even when offline validation metrics improved, the business didn’t see corresponding gains.

2. False positives clustered in specific cohorts
   Legitimate users were flagged at higher rates in particular geographies and customer profiles, suggesting bias or label noise.

The fraud team suspected data drift, but monitoring showed only moderate shifts in feature distributions. The more likely culprit: the training labels themselves.

Northbridge launched a training data audit.

Approach: A Data Audit That Found the Real Bottleneck

1) Auditing the Labels: Measuring Noise, Not Just Quantity

The first step was to sample the training set and compare labels against the best available ground truth: chargeback confirmations, post-transaction investigations, customer support outcomes, and bank return codes. Because fraud is often confirmed after a delay, the team also reviewed whether examples were labeled too early.

The audit revealed two major issues:

• Approximately 18% of examples were mislabeled
  Some “fraud” labels were assigned based on weak signals (e.g., velocity rules) rather than confirmed outcomes. Conversely, some legitimate transactions were labeled as fraud due to unresolved disputes or premature tagging.

• Significant class imbalance
  Like most fintech datasets, true fraud represented a small fraction of transactions. The dataset’s imbalance was not inherently wrong—but it was poorly handled in the labeling pipeline and training setup. In some cohorts, the ratio was skewed even further due to inconsistent capture of fraud outcomes.

This combination created a damaging feedback loop: noisy labels made it harder for the model to learn meaningful patterns, and imbalance made it easier for the model to overfit to the majority class while still producing deceptively “good” accuracy.

2) Mapping the Labeling Pipeline End-to-End

Rather than patching the dataset with one-off fixes, Northbridge treated labeling as a system:

• Label definitions were inconsistent across teams
  “Fraud” meant different things depending on whether a case came from chargebacks, manual review, or automated rules.

• Timing windows were misaligned
  Transactions were labeled before a sufficient confirmation period had elapsed. Some labels flipped later, but the training dataset wasn’t updated.

• Merged data sources introduced conflicts
  When records were joined across systems, mismatched identifiers and partial histories created incorrect outcomes.

Northbridge standardized label definitions and created a single “source of truth” labeling policy that prioritized confirmed outcomes. Where confirmation was uncertain, the team introduced an explicit third state (e.g., “unknown” or “pending”) to avoid forcing noisy binary labels into the training set.

3) Remediating the Dataset: Cleaning, Rebalancing, and Rebuilding

With the problems identified, Northbridge implemented a structured remediation workflow.

Label remediation included:

• Re-labeling the sampled set via a consistent policy
• Backfilling corrected labels for historical records where confirmed outcomes existed
• Excluding ambiguous cases from training until confirmation arrived
• De-duplicating transactions and resolving entity mismatches

Class imbalance remediation included:

• Stratified sampling by key cohorts (region, transaction type, merchant segment)
• Ensuring minority-class examples were sufficiently represented across segments
• Adjusting training weights and sampling strategy to prevent the model from learning “shortcuts” that boosted apparent accuracy but hurt recall on fraud

Notably, the team did not change the model architecture during this phase. This was deliberate: if performance improved, they could attribute gains to the dataset rather than confounding variables.

4) Implementing Ongoing Data Quality Controls

Northbridge treated the remediation as the beginning, not the end. They added controls to prevent regression:

• Label lag policy: only label examples after a minimum confirmation window
• Disagreement checks: flag transactions where sources conflict (e.g., manual review vs. chargeback outcome)
• Noise dashboards: track label flip rates over time
• Cohort integrity checks: monitor class ratios and label quality by segment, not just overall

These controls turned data quality into a measurable, operationalized function rather than a periodic cleanup project.

Results: 23% Accuracy Improvement With No Model Changes

After rebuilding the training dataset and retraining the same model configuration, Northbridge saw a 23% improvement in model accuracy compared to the previous production baseline, with no architectural changes.

While “accuracy” alone is rarely the primary fraud metric, the uplift correlated with meaningful operational improvements:

• Fewer obvious false positives in historically problematic cohorts
• More stable performance across transaction types and regions
• Better alignment between offline validation and production outcomes

Just as importantly, the team regained trust in their experimentation process. Previously, model iterations produced unpredictable results because training labels were inconsistent. With cleaner data, improvements became more attributable—and therefore more repeatable.

(Additional metric changes—such as precision, recall, chargeback rate reduction, or manual review volume—can be expected to move as a consequence of cleaner labels, but specific values will vary by business and were not the focus of this case.)

Why It Worked: The Dataset Became a Better Teacher

A model can only learn what the dataset reliably communicates. In Northbridge’s previous setup:

• Mislabeled examples taught the model that legitimate behavior was fraudulent (and vice versa).
• Imbalance taught the model to optimize for the majority class while missing rare but costly fraud patterns.
• Segment inconsistency taught the model patterns that didn’t generalize.

By remediating the training set, Northbridge reduced contradictory signals and made the data more representative of real-world outcomes. The same model suddenly had a clearer objective—and performed accordingly.

Key Takeaways: What Other Fintech Teams Can Apply

• Treat underperformance as a data problem until proven otherwise.
  If architectural improvements don’t translate into production impact, audit the labels and the pipeline that produces them.

• Quantify label noise.
  Measuring mislabeled rates—even on a sample—can quickly justify prioritizing remediation. In this case, approximately 18% mislabeled examples were enough to cap model performance.

• Standardize what “fraud” means.
  Align definitions across chargebacks, manual review, and rule-based systems. Inconsistent definitions create training contradictions.

• Respect label timing.
  Many fraud outcomes are confirmed after a delay. Labeling too early forces uncertainty into the dataset and creates false ground truth.

• Handle class imbalance by cohort, not just globally.
  Overall ratios can hide pockets of extreme skew. Stratify and validate across segments where fraud behavior differs.

• Build data quality controls into operations.
  Dashboards and automated checks prevent the dataset from degrading silently over time.

Northbridge’s experience underscores a simple lesson: model performance often depends less on model sophistication than on training data integrity. In fraud detection—where labels are delayed, disputed, and operationally messy—clean data is not a “nice to have.” It’s the foundation that determines whether machine learning improves outcomes or merely creates the illusion of progress.