Self-hosted VLM deployment
Deploy vision-language models on your own hardware
If your documents cannot leave your network — regulation, contract, or policy — you need open-weight vision and OCR models running on hardware you control. I size the hardware, pick and quantize the models, stand up the inference server, tune it for your throughput, and validate that the accuracy you get locally is actually acceptable for the job.
- Honest hardware sizing before you buy a single GPU
- Model selection validated on your documents, not on public leaderboards
- vLLM / TensorRT-LLM serving with batching and concurrency tuned to your load
- Works air-gapped, in your VPC, or on a single on-prem workstation
The question that comes before the GPU order
The most expensive mistake in self-hosted AI is buying hardware before validating that an open-weight model can do the job at all. The failure runs both ways: capacity bought for a workload a quantized mid-size model on a single card would have served, or a model chosen for the hardware budget rather than the task, with the shortfall discovered months later. Either way the money is spent before the question is answered.
So the engagement starts backwards from the hardware. First: can an open-weight model hit your accuracy target on your documents? Second: at what quantization does it still hit it? Third: what throughput do you need, and what does that imply for VRAM, card count, and concurrency? Only then does anyone place an order.
That sequence usually saves more than the engagement costs, and occasionally the answer is that self-hosting is not worth it for your workload — which is also a useful result to have in writing.
What the deployment covers
Model selection and evaluation
Candidate open-weight vision and OCR models scored on your documents against ground truth — not on public benchmarks that do not resemble your corpus. You get the comparison table and the reasoning behind the pick.
Quantization and accuracy trade-off
FP16, FP8, AWQ, GPTQ, and GGUF variants scored so you can see exactly what accuracy you give up for what VRAM saving. This is measured per model, not assumed from general guidance.
Hardware sizing
A concrete specification: card model, count, VRAM, host memory, storage throughput, and network. Sized against your measured peak load rather than a vendor's marketing throughput figure.
Inference serving
vLLM or TensorRT-LLM configured for your workload — continuous batching, KV cache sizing, tensor parallelism, prefix caching where it helps — behind an OpenAI-compatible endpoint your existing code can call with a URL change.
Throughput and latency tuning
Benchmarked concurrency curves so you know your real pages-per-second and p95 latency, plus the queueing configuration to keep latency stable under burst.
Operations
Containerised deployment, health checks, GPU and queue metrics, model-update procedure, and a rollback path. Air-gapped installs get an offline artifact bundle.
How it runs
1. Feasibility evaluation
Your documents, a ground-truth set, and three to five candidate open-weight models scored head to head — including against whatever hosted API you would otherwise use, so the accuracy gap is explicit.
2. Sizing report
A written hardware specification with cost, based on the winning model, its quantization, and your measured throughput requirement. Includes the cloud-GPU equivalent so you can compare buy versus rent honestly.
3. Deployment
Inference server stood up on your hardware, integrated behind an OpenAI-compatible endpoint, with your application switched over and running.
4. Tuning and validation
Load testing to your peak, concurrency and batching tuned, and a final accuracy run confirming the deployed system matches the evaluation scores.
5. Handover
Runbook, monitoring dashboards, model-upgrade procedure, and a training session with your infrastructure team.
Typical deployment shapes
Indicative starting points. Actual sizing always comes from the evaluation, because it depends on model choice, quantization, and your concurrency.
| Scenario | Typical shape | What it suits |
|---|---|---|
| Single-workstation pilot | One 48GB-class card, quantized mid-size VLM | Proving the workflow before capital spend |
| Departmental production | 2–4 cards, batched serving, queue in front | Steady internal document volume with burst tolerance |
| High-throughput pipeline | Multi-node, tensor parallel, autoscaled workers | Continuous ingestion at scale |
| Air-gapped / classified | Offline artifact bundle, no external network path | Environments where no egress is permitted |
| Private cloud (VPC) | Managed GPU instances in your own account | Data-residency requirements without owning hardware |
You should consider self-hosting if
- Regulation or client contracts prohibit sending documents to third-party APIs
- Your data-residency requirements are not met by available API regions
- Per-document API cost at your volume now exceeds amortised hardware cost
- You need deterministic model versions that do not change under you
- Latency requirements rule out a round trip to an external provider
- You already own idle GPU capacity and want to use it
Engagement options
Feasibility & sizing
from $1,800
2–3 weeks
Find out whether self-hosting works for your workload — and what it would cost — before committing to hardware.
- Ground-truth set from your documents
- 3–5 open-weight models scored head to head
- Quantization accuracy trade-off table
- Hardware specification with cost
- Buy-versus-rent comparison
Full deployment
from $3,000
3–8 weeks
Models running in production on your hardware, integrated with your application.
- Everything in feasibility & sizing
- vLLM / TensorRT-LLM serving setup
- OpenAI-compatible endpoint integration
- Throughput and latency tuning
- Monitoring, runbook, and team training
Ongoing operation
from $1,800 / month
Rolling
Model upgrades, capacity planning, and performance work as your volume grows.
- Model evaluation and upgrade as new weights release
- Capacity planning against observed growth
- Performance regression monitoring
- Priority support on inference incidents
Frequently asked questions
Which models do you deploy?
Open-weight vision-language models in the Qwen-VL and InternVL families, open OCR models, and open text models for the normalization step. The specific pick comes out of the evaluation on your documents — model rankings change every few months and a recommendation made without measuring yours is worth very little.
Will self-hosted accuracy match a frontier API?
Sometimes, and sometimes not. On structured document extraction the gap is often smaller than people expect, especially once domain validation rules are layered on top. The evaluation gives you the actual number for your documents so the decision is made on data. If the gap is unacceptable I will tell you that.
Can you work with hardware we already own?
Yes. If you have existing GPU capacity the evaluation targets it directly, and the sizing report tells you whether it is sufficient, needs augmenting, or is genuinely the wrong shape for the workload.
Do you support fully air-gapped environments?
Yes. Deliverables include an offline artifact bundle — model weights, container images, and dependencies — with an installation procedure that requires no external network access. Access arrangements for the install itself are agreed per engagement.
What ongoing maintenance does this need?
Less than most teams fear, but not zero. Realistically: GPU driver and container updates, monitoring, and a model evaluation every few months as better open weights release. That is a few days a quarter for a competent infrastructure engineer, or a retainer if you would rather not own it.
Find out if self-hosting is viable before you buy hardware
Send me a sample of your documents and your throughput requirement. The feasibility evaluation answers the question with numbers, and it is scoped so you can stop there if the answer is no.
Related
On-premise & private cloud
On-premise and private-cloud document AI deployment
Deploy the full document extraction stack inside your own infrastructure — VPC, private cloud, or air-gapped — with no document leaving your network. Sizing, deployment, and compliance documentation.
Custom OCR engineering
Custom OCR development for documents that off-the-shelf OCR gets wrong
Custom OCR pipeline development for scanned, photographed, and low-quality documents — layout-aware extraction, confidence scoring, vision-model escalation, and ground-truth accuracy benchmarks.
RAG systems
RAG systems that retrieve the right thing — and prove it
Design, build, and evaluate retrieval-augmented generation systems: chunking and indexing strategy, hybrid and reranked retrieval, grounded answer generation, and retrieval evaluation harnesses.
Enterprise
Enterprise bank statement processing at volume
High-volume statement and document processing for finance, lending, and accounting teams — dedicated infrastructure, data-residency options, SLAs, custom schemas, and direct engineering access.
This page is also available as markdown for AI agents: /services/self-hosted-vision-language-models.md · index at /llms.txt. Canonical URL: https://parsemystatement.com/services/self-hosted-vision-language-models.