# vLLM Backend Plan and Test Results

Date: 2026-06-09

## Plan

1. Keep the legacy OCR endpoint at `/v1/api/ai/suya_ocr/` for baseline comparison.
2. Add `/v1/api/ai/suya_ocr_vllm/`, backed by Surya2 `SuryaInferenceManager(method="vllm")`.
3. Use the Surya vLLM backend settings from the referenced Surya checkout:
   - `SURYA_INFERENCE_BACKEND=vllm`
   - `SURYA_INFERENCE_KEEP_ALIVE=true`
   - T4 default: `VLLM_GPU_TYPE=t4`, `VLLM_DTYPE=float16`, `SURYA_INFERENCE_PARALLEL=16`
   - optional tuning through `VLLM_GPUS`, `VLLM_GPU_MEMORY_UTILIZATION`, and `SURYA_INFERENCE_URL`
4. Add a concurrency benchmark that sends the same payload to the legacy and vLLM endpoints and writes latency/throughput JSON.
5. Verify syntax locally, then run the benchmark on a host with the Surya2 package, Docker, and NVIDIA GPU access.

## Implemented Files

- `vllm_tools.py`: vLLM-backed Surya2 predictor setup and OCR response formatting.
- `api.py`: new `/v1/api/ai/suya_ocr_vllm/` endpoint and health endpoint.
- `concurrency_test.py`: repeatable old-vs-new endpoint concurrency benchmark.
- `concurrency_test_results.json`: placeholder result file that the benchmark overwrites with measured results.
- `requirements.txt`: dependency alignment for Surya2/vLLM OpenAI client support.
- `Dockerfile`: T4 vLLM defaults. The referenced backend maps T4 16 GB VRAM to `max_num_seqs=16` and `max_num_batched_tokens=4096`; `float16` is required because T4 does not support `bfloat16`.

## Single-Container Kubernetes Deployment

The deployment now uses `vllm/vllm-openai:v0.20.1` as the base image. The container starts the vLLM OpenAI-compatible server in the background and runs `api.py` in the foreground through `scripts/start_single_container.sh`.

The API no longer depends on nested Docker. It attaches to:

```text
SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1
```

The Surya source needed by the API is vendored into this repo under `surya/`, so the container does not depend on `/path/to/surya` at runtime.

Serving defaults for T4:

```text
SUYA_OCR_MODE=block
SUYA_MAX_BATCH_SIZE=8
SUYA_BATCH_WAIT_MS=25
SURYA_INFERENCE_PARALLEL=8
SURYA_INFERENCE_MAX_RETRIES=1
SURYA_MAX_BLOCKS_PER_PAGE=80
VLLM_DTYPE=float16
VLLM_MAX_NUM_SEQS=16
VLLM_MAX_BATCHED_TOKENS=4096
```

Speed-related changes:

- `/v1/api/ai/suya_ocr_vllm/` uses `vllm_batcher.py` to coalesce concurrent HTTP requests into batch Surya calls.
- `vllm_tools.py` batches layout and recognition across images.
- Default OCR mode is block mode, avoiding long full-page generations and hidden fallback bursts.
- Vendored Surya caps block OCR per page with `SURYA_MAX_BLOCKS_PER_PAGE`.
- Retry amplification is limited with `SURYA_INFERENCE_MAX_RETRIES=1`.
- The endpoint skips drawing annotated images because the API response only returns JSON/text.
- Base64 input is decoded once per request and the same PIL image object is reused for low/high-res OCR paths unless a caller explicitly asks for a copy.
- vLLM image payload encoding defaults to high-quality JPEG (`SUYA_VLLM_IMAGE_FORMAT=JPEG`, `SUYA_VLLM_JPEG_QUALITY=92`) to reduce CPU serialization and local vLLM transfer size versus PNG. Set `SUYA_VLLM_IMAGE_FORMAT=PNG` if lossless transport is required.

## Benchmark Command

```bash
python concurrency_test.py \
  --image /path/to/surya/ttt.png \
  --requests 20 \
  --concurrency 8 \
  --output concurrency_test_results.json
```

## Current Test Results

Import verification passed with `PYTHONPATH=/path/to/surya`.

Concurrency benchmark run:

```bash
/opt/conda/bin/python concurrency_test.py \
  --image temp_image_c3c56948-282e-453b-8fac-6c482243d1e5.jpg \
  --requests 16 \
  --concurrency 16 \
  --timeout 900 \
  --output concurrency_test_results.json
```

Run started at `2026-06-11T14:35:17+0400`. Docker NVIDIA runtime and GPU checks passed. The local Python environment does not include the `vllm` package, so this validation attached the updated API to the already-warm local `vllm/vllm-openai:v0.20.1` server on port `43401` with `SURYA_INFERENCE_URL=http://127.0.0.1:43401/v1`. The API path still used the single-container attach mode (`SURYA_INFERENCE_AUTOSTART=false`) and did not start nested Docker.

| Endpoint | Requests | Concurrency | Success | Failed | Wall seconds | Throughput rps | Mean latency | P50 latency | P95 latency | Max latency |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `/v1/api/ai/suya_ocr/` | 16 | 16 | 16 | 0 | 43.62 | 0.3668 | 40.81 | 41.84 | 43.39 | 43.62 |
| `/v1/api/ai/suya_ocr_vllm/` | 16 | 16 | 16 | 0 | 83.59 | 0.1914 | 59.08 | 60.41 | 83.52 | 83.54 |

Latency values are per request in seconds. Mean latency is the average request elapsed time, not the total benchmark wall time.

## Approach A Outcome (2026-06-12)

Diagnosis + work-reduction tuning (see `docs/diagnosis_baseline.md` and
`docs/tier2_results.md`). Measured on the A100 dev host against the warm
`surya-vllm-43401` server — same host/server as the table above, so comparable.
**Production is T4; the fix below is hardware-independent, the absolute numbers
are not.**

**Root cause:** the request batcher coalesced requests then dispatched them only
`SURYA_INFERENCE_PARALLEL=8` wide, so only 8 of vLLM's 16 sequence slots were
used (poller `peak_running=8`). The GPU sat half-idle, which is why the vLLM
endpoint was *slower* than the flooding legacy path.

**Fix (Tier 1):** scale the worker pool to the in-flight block count, capped by a
new `SURYA_INFERENCE_MAX_INFLIGHT` (default 16, = `VLLM_MAX_NUM_SEQS`). This fills
all 16 slots (`peak_running=16`).

| vLLM endpoint, 16/16 | mean | p50 | p95 | peak_running |
| --- | ---: | ---: | ---: | ---: |
| MI=8 (before) | 50.22 | 49.48 | 72.69 | 8 |
| MI=16 (fix) | 33.75 | 36.11 | 49.15 | 16 |

- Mean −33%, p95 −32%; single request 6.16 s → ~5.3 s. CER divergence 0.0009
  (lossless; gate ≤ 0.005). The vLLM endpoint now beats legacy on mean and median.
- MI=24 overcommits (worse). Batch-size / batch-wait are within noise.

**Tier 2 (token ceiling, recognition DPI, block cap):** no latency benefit on
this sparse page — decode dominates and the page is light — so all Tier-2 knobs
are left at defaults. They remain relevant for dense / pathological pages.

**Locked change:** `SURYA_INFERENCE_MAX_INFLIGHT=16` added to `Dockerfile` and
`scripts/start_single_container.sh`. Re-confirm absolute latencies on a T4 before
treating them as production SLAs; the regression fix itself applies unchanged.

## FP8 Experiment

I attempted to start the latest vLLM image with multimodal encoder FP8 enabled:

```bash
--dtype bfloat16 \
--mm-encoder-attn-dtype fp8
```

The first attempt on `cuda:0` failed before serving traffic because GPU 0 did not have enough free memory for the default `--gpu-memory-utilization 0.85` target.

The second attempt on `cuda:1` booted farther, but vLLM aborted during model initialization with:

```text
ValueError: mm_encoder_attn_dtype='fp8' requires the FlashInfer cuDNN backend with cuDNN >= 9.17.1 on a GPU with native FP8 support.
```

That means:

- no FP8 latency number was produced
- the current A100 hardware is not a valid target for this exact FP8 path
- the failure is architectural, not a tuning issue

To measure a real FP8 speed change, the next engineer should use a GPU with native FP8 support and a vLLM build that satisfies the FlashInfer cuDNN requirement for multimodal encoder FP8.

## NVIDIA Docker Fix

Docker currently has no registered `nvidia` runtime. The expected daemon config is:

```json
{
  "data-root": "/data/docker_cache",
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}
```

This user cannot write `/etc/docker/daemon.json` or restart Docker because `sudo` requires a password. The host does support Docker GPU passthrough through modern `--gpus all`, so the API now patches Surya's vLLM spawn path to use `scripts/docker_nvidia_runtime_compat.sh`. That wrapper strips the legacy `--runtime nvidia` pair and preserves `--gpus device=...`.

An admin can still register the named runtime permanently with:

```bash
sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json
sudo systemctl restart docker
docker run --rm --runtime nvidia --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```

The repo includes `scripts/configure_nvidia_docker.sh` with those commands.
