ParseMyStatement v2.0 — Block-Level Confidence Sounded Great. Our Own Benchmark Caught the Regression.

ParseMyStatement v2.0 — Block-level confidence (include_blocks=True) sounded great. Our own benchmark caught the regression.

This is the most honest post in the series. Our v1.1 release pushed mean transaction F1 to 93.7%. In July 2026 we shipped ParseMyStatement v2.0, a headline feature release: we turned on Mistral's block-level metadata (include_blocks=True) and per-page confidence scores. The marketing instinct was to celebrate — more structure, more transparency, a confidence score for every page. Then we ran the same three real statements through the same harness, and mean transaction F1 fell from 93.7% to 51.6%.

This is the story of how a rigorous, ground-truth benchmark — the exact one we encourage every API buyer to demand — stopped us from shipping a regression to production finance workflows.

What v2.0 changed under the hood

We added two parameters to the Mistral OCR call and switched normalization from gpt-4o-mini to deepseek-ai/DeepSeek-V4-Flash on DeepInfra:

response = client.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "document_url", "document_url": pdf_url},
    include_blocks=True,                  # NEW in v2.0
    confidence_scores_granularity="page", # NEW in v2.0
)

include_blocks=True returns typed blocks (title, table, header, footer…) with bounding boxes. We exposed them as ocr_metadata on the result, flagged pages with average confidence below 0.7 in low_confidence_pages, and issued a human-readable warning.

{
  "ocr_metadata": {
    "0": {
      "blocks": [
        { "type": "title", "content": "## HSBC Live+ Credit Card" },
        { "type": "table", "content": "<table>...</table>", "table_id": "tbl-0.html" }
      ],
      "confidence_scores": { "average_page_confidence_score": 0.9842, "minimum_page_confidence_score": 0.1937 }
    }
  },
  "low_confidence_pages": [5]
}

The v2.0 regression, measured

Same harness, same three real statements, same hand-written ground truth:

Transaction F1 by statement (v2.0, Blocks+Confidence):
Canara  ███████████████████████   81.6%
HSBC    ██████████████████████  73.3%
ICICI                               0.0%  ← bank detected, zero rows extracted

The worst signal was ICICI: v2.0 detected the bank but extracted zero transactions. The new block metadata path — which we built "to make headers and footers filterable" — ended up disrupting the very table structure that OCR-only mode had let flow through cleanly in v1.1. The confidence payload changed the shape of the OCR response, and the normalization model (deepseek-ai/DeepSeek-V4-Flash) saw a different, noisier structure.

Why the benchmark saved us

Here is the uncomfortable part worth being honest about: the visibility numbers looked great. We could show confidence scores, warning lists, and block metadata — genuinely useful for debugging. Had we shipped on vibe, we would have told the world v2.0 was a leap forward. The ground-truth harness said otherwise within minutes. Because we score against the banks' own exports, not against how confident we feel, a regression could not hide behind pretty metadata.

This is exactly why our v1.0 methodology post insists on publishing model names and real F1. When the numbers move the wrong way, you want a test that yells at you — and it did.

The fix and the lesson

We did not ship v2.0's blocks+confidence path as the default. The metadata stayed available for debugging, but the pipeline reverted to the proven table-aware path, now on deepseek-ai/DeepSeek-V4-Flash. What came next — hard prompt hardening and tolerant JSON parsing — is the release that finally beat our own record. That is v2.1.

The lesson we now apply to every release, and the one we want every team buying an OCR API to apply too:

A benchmark that can say "no" is worth more than a feature that can say "yes." Publish your model names, publish your ground truth, and let honest F1 drive your roadmap.

Try it

The current production engine — with include_blocks confidence available and hardened normalization — is live on our home page and the developer API. See how the versions compare end-to-end in our model history comparison.