Bank Statement OCR Model History & Comparison: ParseMyStatement from v1.0 to v2.1

Bank Statement OCR Model History & Comparison — ParseMyStatement from v1.0 to v2.1 in F1, recall, and real statements

This is the fifth and final post in our OCR series — and the one that tells the whole story. We built ParseMyStatement to turn ugly bank statement PDFs into clean CSV, Excel, and JSON, and over five months the extraction engine went from 90.5% to 97.7% mean transaction F1 — with 100% F1 on two of our three test statements. This post is an overall comparison of every model we've shipped, the growth of the product, the metrics that matter, and the honest lessons that came out of a real, ground-truth benchmark.

If you are evaluating any bank statement OCR API, this is the transparency bar we hold ourselves to: model names, real statements, reproducible methodology.

The three real statements we score on

Every release in this series is measured on the same three real personal statements. Ground truth comes from the banks' own CSV/XLS exports — not synthetic PDFs:

That is 699 real transactions across a page-heavy savings account, a multi-line HSBC statement, and a dense, sign-less credit card bill — the three layouts that break most OCR tools.

The full model history, side by side

Mean Transaction F1 across the release history:
v1.0 (Mar)     ███████████████████████████  90.5%
v1.1 (Jun)     ████████████████████████████  93.7%
v2.0 (Jul)     ███████████████  51.6%   ← regression, caught by benchmark
v2.1 (Aug)     █████████████████████████████  97.7%

Mean overall score across the release history:
v1.0 (Mar)     ████████████████  53.8%
v1.1 (Jun)     ██████████████████  62.4%
v2.0 (Jul)     █████████████  46.1%
v2.1 (Aug)     ████████████████████  68.4%

Per-statement F1 at the latest release (v2.1)

*Credit card statements have no running-balance column, so balance accuracy is not scored there.

The growth story the numbers hide

Mean F1 is the headline, but four quieter trends define the product's growth:

1. Date accuracy has been ~99%+ since v1.1 and stayed there through the regression. Dates are the column that silently destroys spreadsheets — day/month swaps, missing years, UTC drift. This is the metric your bookkeeper cares about most, and it is now effectively solved across all layouts.

2. Debit/credit accuracy is where stability was earned. Canara and ICICI both hit 100% debit accuracy in v2.1. The credit card — which read 14.8% debit accuracy in v1.0 and 0% in v2.0 — is now flawless, proof that the hardened tabular path solved the hardest sign-less layout.

3. Bank detection went 0% → 100%. v1.0 didn't even score it correctly; every release since bank-detects 100% on all three statements. Bank detection is the cheap, high-value win few OCR vendors publish.

4. Overall score is a stricter, more honest number than F1. Our overall score folds in description similarity, reference accuracy, and balance validation — the fields real reconciliation teams actually audit. It climbed 53.8% → 68.4%, and it is the metric we advertise conservatively so the more aggressive F1 is always an upside, never a stretch.

What we learned — the honest takeaways

• An LLM normalization stage can hurt. v1.1 (OCR-only) proved that reading a clean table directly beats letting a model reinterpret it. v2.1 found the balance: keep the model, but harden it.
• A benchmark that says no is your best feature. v2.0's blocks+confidence looked great and scored terribly (51.6%). Ground truth caught it before production finance got it. This is why we publish real F1 and real statements, not confidence-score marketing.
• Precision and recall trade off per layout. HSBC's multi-line rows remain the hardest layout: our extractor still deviates from the corrected ground truth there (365 rows extracted vs 382). There is no single free lunch across every bank — which is precisely why you measure per statement, not per marketing slide.

The product behind the model

This OCR journey powers a full product: drag-and-drop conversion on the home page, a developer API with named keys (psm_...), batch upload, balance-checked exports, JSON/CSV/XLSX output, and 57+ free financial tools at /tools. Every conversion also records anonymized production stats (zero PII) that feed a live accuracy dashboard — the same discipline that caught the v2.0 regression applies to every file that runs through us.

Try the current engine

Upload any bank or credit card statement on the home page — no signup to try — or read the per-release stories in the rest of this series:

• v1.0 — Mistral OCR launch
• v1.1 — OCR-only mode
• v2.0 — the regression we caught
• v2.1 — the record