surya-ocr/IMPLEMENTATION_PLAN_AND_TEST_RESULTS.md

vLLM Backend Plan and Test Results

Date: 2026-06-09

Plan

1. Keep the legacy OCR endpoint at /v1/api/ai/suya_ocr/ for baseline comparison.
2. Add /v1/api/ai/suya_ocr_vllm/, backed by Surya2 SuryaInferenceManager(method="vllm").
3. Use the Surya vLLM backend settings from the referenced Surya checkout:
   • SURYA_INFERENCE_BACKEND=vllm
   • SURYA_INFERENCE_KEEP_ALIVE=true
   • T4 default: VLLM_GPU_TYPE=t4, VLLM_DTYPE=float16, SURYA_INFERENCE_PARALLEL=16
   • optional tuning through VLLM_GPUS, VLLM_GPU_MEMORY_UTILIZATION, and SURYA_INFERENCE_URL
4. Add a concurrency benchmark that sends the same payload to the legacy and vLLM endpoints and writes latency/throughput JSON.
5. Verify syntax locally, then run the benchmark on a host with the Surya2 package, Docker, and NVIDIA GPU access.

Implemented Files

• vllm_tools.py: vLLM-backed Surya2 predictor setup and OCR response formatting.
• api.py: new /v1/api/ai/suya_ocr_vllm/ endpoint and health endpoint.
• concurrency_test.py: repeatable old-vs-new endpoint concurrency benchmark.
• concurrency_test_results.json: placeholder result file that the benchmark overwrites with measured results.
• requirements.txt: dependency alignment for Surya2/vLLM OpenAI client support.
• Dockerfile: T4 vLLM defaults. The referenced backend maps T4 16 GB VRAM to max_num_seqs=16 and max_num_batched_tokens=4096; float16 is required because T4 does not support bfloat16.

Single-Container Kubernetes Deployment

The deployment now uses vllm/vllm-openai:v0.20.1 as the base image. The container starts the vLLM OpenAI-compatible server in the background and runs api.py in the foreground through scripts/start_single_container.sh.

The API no longer depends on nested Docker. It attaches to:

SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1

The Surya source needed by the API is vendored into this repo under surya/, so the container does not depend on /path/to/surya at runtime.

Serving defaults for T4:

SUYA_OCR_MODE=block
SUYA_MAX_BATCH_SIZE=8
SUYA_BATCH_WAIT_MS=25
SURYA_INFERENCE_PARALLEL=8
SURYA_INFERENCE_MAX_RETRIES=1
SURYA_MAX_BLOCKS_PER_PAGE=80
VLLM_DTYPE=float16
VLLM_MAX_NUM_SEQS=16
VLLM_MAX_BATCHED_TOKENS=4096

Speed-related changes:

• /v1/api/ai/suya_ocr_vllm/ uses vllm_batcher.py to coalesce concurrent HTTP requests into batch Surya calls.
• vllm_tools.py batches layout and recognition across images.
• Default OCR mode is block mode, avoiding long full-page generations and hidden fallback bursts.
• Vendored Surya caps block OCR per page with SURYA_MAX_BLOCKS_PER_PAGE.
• Retry amplification is limited with SURYA_INFERENCE_MAX_RETRIES=1.
• The endpoint skips drawing annotated images because the API response only returns JSON/text.
• Base64 input is decoded once per request and the same PIL image object is reused for low/high-res OCR paths unless a caller explicitly asks for a copy.
• vLLM image payload encoding defaults to high-quality JPEG (SUYA_VLLM_IMAGE_FORMAT=JPEG, SUYA_VLLM_JPEG_QUALITY=92) to reduce CPU serialization and local vLLM transfer size versus PNG. Set SUYA_VLLM_IMAGE_FORMAT=PNG if lossless transport is required.

Benchmark Command

python concurrency_test.py \
  --image /path/to/surya/ttt.png \
  --requests 20 \
  --concurrency 8 \
  --output concurrency_test_results.json

Current Test Results

Import verification passed with PYTHONPATH=/path/to/surya.

Concurrency benchmark run:

/opt/conda/bin/python concurrency_test.py \
  --image temp_image_c3c56948-282e-453b-8fac-6c482243d1e5.jpg \
  --requests 16 \
  --concurrency 16 \
  --timeout 900 \
  --output concurrency_test_results.json

Run started at 2026-06-11T14:35:17+0400. Docker NVIDIA runtime and GPU checks passed. The local Python environment does not include the vllm package, so this validation attached the updated API to the already-warm local vllm/vllm-openai:v0.20.1 server on port 43401 with SURYA_INFERENCE_URL=http://127.0.0.1:43401/v1. The API path still used the single-container attach mode (SURYA_INFERENCE_AUTOSTART=false) and did not start nested Docker.

Endpoint	Requests	Concurrency	Success	Failed	Wall seconds	Throughput rps	Mean latency	P50 latency	P95 latency	Max latency
/v1/api/ai/suya_ocr/	16	16	16	0	43.62	0.3668	40.81	41.84	43.39	43.62
/v1/api/ai/suya_ocr_vllm/	16	16	16	0	83.59	0.1914	59.08	60.41	83.52	83.54

Latency values are per request in seconds. Mean latency is the average request elapsed time, not the total benchmark wall time.

Approach A Outcome (2026-06-12)

Diagnosis + work-reduction tuning (see docs/diagnosis_baseline.md and docs/tier2_results.md). Measured on the A100 dev host against the warm surya-vllm-43401 server — same host/server as the table above, so comparable. Production is T4; the fix below is hardware-independent, the absolute numbers are not.

Root cause: the request batcher coalesced requests then dispatched them only SURYA_INFERENCE_PARALLEL=8 wide, so only 8 of vLLM's 16 sequence slots were used (poller peak_running=8). The GPU sat half-idle, which is why the vLLM endpoint was slower than the flooding legacy path.

Fix (Tier 1): scale the worker pool to the in-flight block count, capped by a new SURYA_INFERENCE_MAX_INFLIGHT (default 16, = VLLM_MAX_NUM_SEQS). This fills all 16 slots (peak_running=16).

vLLM endpoint, 16/16	mean	p50	p95	peak_running
MI=8 (before)	50.22	49.48	72.69	8
MI=16 (fix)	33.75	36.11	49.15	16

• Mean −33%, p95 −32%; single request 6.16 s → ~5.3 s. CER divergence 0.0009 (lossless; gate ≤ 0.005). The vLLM endpoint now beats legacy on mean and median.
• MI=24 overcommits (worse). Batch-size / batch-wait are within noise.

Tier 2 (token ceiling, recognition DPI, block cap): no latency benefit on this sparse page — decode dominates and the page is light — so all Tier-2 knobs are left at defaults. They remain relevant for dense / pathological pages.

Locked change: SURYA_INFERENCE_MAX_INFLIGHT=16 added to Dockerfile and scripts/start_single_container.sh. Re-confirm absolute latencies on a T4 before treating them as production SLAs; the regression fix itself applies unchanged.

FP8 Experiment

I attempted to start the latest vLLM image with multimodal encoder FP8 enabled:

--dtype bfloat16 \
--mm-encoder-attn-dtype fp8

The first attempt on cuda:0 failed before serving traffic because GPU 0 did not have enough free memory for the default --gpu-memory-utilization 0.85 target.

The second attempt on cuda:1 booted farther, but vLLM aborted during model initialization with:

ValueError: mm_encoder_attn_dtype='fp8' requires the FlashInfer cuDNN backend with cuDNN >= 9.17.1 on a GPU with native FP8 support.

That means:

• no FP8 latency number was produced
• the current A100 hardware is not a valid target for this exact FP8 path
• the failure is architectural, not a tuning issue

To measure a real FP8 speed change, the next engineer should use a GPU with native FP8 support and a vLLM build that satisfies the FlashInfer cuDNN requirement for multimodal encoder FP8.

NVIDIA Docker Fix

Docker currently has no registered nvidia runtime. The expected daemon config is:

{
  "data-root": "/data/docker_cache",
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}

This user cannot write /etc/docker/daemon.json or restart Docker because sudo requires a password. The host does support Docker GPU passthrough through modern --gpus all, so the API now patches Surya's vLLM spawn path to use scripts/docker_nvidia_runtime_compat.sh. That wrapper strips the legacy --runtime nvidia pair and preserves --gpus device=....

An admin can still register the named runtime permanently with:

sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json
sudo systemctl restart docker
docker run --rm --runtime nvidia --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

The repo includes scripts/configure_nvidia_docker.sh with those commands.