Fu DaiandClaude Opus 4.8
1a585693be
FastAPI service wrapping the Surya-OCR-2 model (datalab-to) served through vLLM: legacy /v1/api/ai/* endpoints, an OpenAI-compatible /v1/chat/completions endpoint, a coalescing request batcher, a local OCR CLI, Docker packaging, multilingual example outputs, and quantization/concurrency benchmarks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
183 lines
8.2 KiB
Markdown
183 lines
8.2 KiB
Markdown
# vLLM Backend Plan and Test Results
|
||
|
||
Date: 2026-06-09
|
||
|
||
## Plan
|
||
|
||
1. Keep the legacy OCR endpoint at `/v1/api/ai/suya_ocr/` for baseline comparison.
|
||
2. Add `/v1/api/ai/suya_ocr_vllm/`, backed by Surya2 `SuryaInferenceManager(method="vllm")`.
|
||
3. Use the Surya vLLM backend settings from the referenced Surya checkout:
|
||
- `SURYA_INFERENCE_BACKEND=vllm`
|
||
- `SURYA_INFERENCE_KEEP_ALIVE=true`
|
||
- T4 default: `VLLM_GPU_TYPE=t4`, `VLLM_DTYPE=float16`, `SURYA_INFERENCE_PARALLEL=16`
|
||
- optional tuning through `VLLM_GPUS`, `VLLM_GPU_MEMORY_UTILIZATION`, and `SURYA_INFERENCE_URL`
|
||
4. Add a concurrency benchmark that sends the same payload to the legacy and vLLM endpoints and writes latency/throughput JSON.
|
||
5. Verify syntax locally, then run the benchmark on a host with the Surya2 package, Docker, and NVIDIA GPU access.
|
||
|
||
## Implemented Files
|
||
|
||
- `vllm_tools.py`: vLLM-backed Surya2 predictor setup and OCR response formatting.
|
||
- `api.py`: new `/v1/api/ai/suya_ocr_vllm/` endpoint and health endpoint.
|
||
- `concurrency_test.py`: repeatable old-vs-new endpoint concurrency benchmark.
|
||
- `concurrency_test_results.json`: placeholder result file that the benchmark overwrites with measured results.
|
||
- `requirements.txt`: dependency alignment for Surya2/vLLM OpenAI client support.
|
||
- `Dockerfile`: T4 vLLM defaults. The referenced backend maps T4 16 GB VRAM to `max_num_seqs=16` and `max_num_batched_tokens=4096`; `float16` is required because T4 does not support `bfloat16`.
|
||
|
||
## Single-Container Kubernetes Deployment
|
||
|
||
The deployment now uses `vllm/vllm-openai:v0.20.1` as the base image. The container starts the vLLM OpenAI-compatible server in the background and runs `api.py` in the foreground through `scripts/start_single_container.sh`.
|
||
|
||
The API no longer depends on nested Docker. It attaches to:
|
||
|
||
```text
|
||
SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1
|
||
```
|
||
|
||
The Surya source needed by the API is vendored into this repo under `surya/`, so the container does not depend on `/path/to/surya` at runtime.
|
||
|
||
Serving defaults for T4:
|
||
|
||
```text
|
||
SUYA_OCR_MODE=block
|
||
SUYA_MAX_BATCH_SIZE=8
|
||
SUYA_BATCH_WAIT_MS=25
|
||
SURYA_INFERENCE_PARALLEL=8
|
||
SURYA_INFERENCE_MAX_RETRIES=1
|
||
SURYA_MAX_BLOCKS_PER_PAGE=80
|
||
VLLM_DTYPE=float16
|
||
VLLM_MAX_NUM_SEQS=16
|
||
VLLM_MAX_BATCHED_TOKENS=4096
|
||
```
|
||
|
||
Speed-related changes:
|
||
|
||
- `/v1/api/ai/suya_ocr_vllm/` uses `vllm_batcher.py` to coalesce concurrent HTTP requests into batch Surya calls.
|
||
- `vllm_tools.py` batches layout and recognition across images.
|
||
- Default OCR mode is block mode, avoiding long full-page generations and hidden fallback bursts.
|
||
- Vendored Surya caps block OCR per page with `SURYA_MAX_BLOCKS_PER_PAGE`.
|
||
- Retry amplification is limited with `SURYA_INFERENCE_MAX_RETRIES=1`.
|
||
- The endpoint skips drawing annotated images because the API response only returns JSON/text.
|
||
- Base64 input is decoded once per request and the same PIL image object is reused for low/high-res OCR paths unless a caller explicitly asks for a copy.
|
||
- vLLM image payload encoding defaults to high-quality JPEG (`SUYA_VLLM_IMAGE_FORMAT=JPEG`, `SUYA_VLLM_JPEG_QUALITY=92`) to reduce CPU serialization and local vLLM transfer size versus PNG. Set `SUYA_VLLM_IMAGE_FORMAT=PNG` if lossless transport is required.
|
||
|
||
## Benchmark Command
|
||
|
||
```bash
|
||
python concurrency_test.py \
|
||
--image /path/to/surya/ttt.png \
|
||
--requests 20 \
|
||
--concurrency 8 \
|
||
--output concurrency_test_results.json
|
||
```
|
||
|
||
## Current Test Results
|
||
|
||
Import verification passed with `PYTHONPATH=/path/to/surya`.
|
||
|
||
Concurrency benchmark run:
|
||
|
||
```bash
|
||
/opt/conda/bin/python concurrency_test.py \
|
||
--image temp_image_c3c56948-282e-453b-8fac-6c482243d1e5.jpg \
|
||
--requests 16 \
|
||
--concurrency 16 \
|
||
--timeout 900 \
|
||
--output concurrency_test_results.json
|
||
```
|
||
|
||
Run started at `2026-06-11T14:35:17+0400`. Docker NVIDIA runtime and GPU checks passed. The local Python environment does not include the `vllm` package, so this validation attached the updated API to the already-warm local `vllm/vllm-openai:v0.20.1` server on port `43401` with `SURYA_INFERENCE_URL=http://127.0.0.1:43401/v1`. The API path still used the single-container attach mode (`SURYA_INFERENCE_AUTOSTART=false`) and did not start nested Docker.
|
||
|
||
| Endpoint | Requests | Concurrency | Success | Failed | Wall seconds | Throughput rps | Mean latency | P50 latency | P95 latency | Max latency |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| `/v1/api/ai/suya_ocr/` | 16 | 16 | 16 | 0 | 43.62 | 0.3668 | 40.81 | 41.84 | 43.39 | 43.62 |
|
||
| `/v1/api/ai/suya_ocr_vllm/` | 16 | 16 | 16 | 0 | 83.59 | 0.1914 | 59.08 | 60.41 | 83.52 | 83.54 |
|
||
|
||
Latency values are per request in seconds. Mean latency is the average request elapsed time, not the total benchmark wall time.
|
||
|
||
## Approach A Outcome (2026-06-12)
|
||
|
||
Diagnosis + work-reduction tuning (see `docs/diagnosis_baseline.md` and
|
||
`docs/tier2_results.md`). Measured on the A100 dev host against the warm
|
||
`surya-vllm-43401` server — same host/server as the table above, so comparable.
|
||
**Production is T4; the fix below is hardware-independent, the absolute numbers
|
||
are not.**
|
||
|
||
**Root cause:** the request batcher coalesced requests then dispatched them only
|
||
`SURYA_INFERENCE_PARALLEL=8` wide, so only 8 of vLLM's 16 sequence slots were
|
||
used (poller `peak_running=8`). The GPU sat half-idle, which is why the vLLM
|
||
endpoint was *slower* than the flooding legacy path.
|
||
|
||
**Fix (Tier 1):** scale the worker pool to the in-flight block count, capped by a
|
||
new `SURYA_INFERENCE_MAX_INFLIGHT` (default 16, = `VLLM_MAX_NUM_SEQS`). This fills
|
||
all 16 slots (`peak_running=16`).
|
||
|
||
| vLLM endpoint, 16/16 | mean | p50 | p95 | peak_running |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| MI=8 (before) | 50.22 | 49.48 | 72.69 | 8 |
|
||
| MI=16 (fix) | 33.75 | 36.11 | 49.15 | 16 |
|
||
|
||
- Mean −33%, p95 −32%; single request 6.16 s → ~5.3 s. CER divergence 0.0009
|
||
(lossless; gate ≤ 0.005). The vLLM endpoint now beats legacy on mean and median.
|
||
- MI=24 overcommits (worse). Batch-size / batch-wait are within noise.
|
||
|
||
**Tier 2 (token ceiling, recognition DPI, block cap):** no latency benefit on
|
||
this sparse page — decode dominates and the page is light — so all Tier-2 knobs
|
||
are left at defaults. They remain relevant for dense / pathological pages.
|
||
|
||
**Locked change:** `SURYA_INFERENCE_MAX_INFLIGHT=16` added to `Dockerfile` and
|
||
`scripts/start_single_container.sh`. Re-confirm absolute latencies on a T4 before
|
||
treating them as production SLAs; the regression fix itself applies unchanged.
|
||
|
||
## FP8 Experiment
|
||
|
||
I attempted to start the latest vLLM image with multimodal encoder FP8 enabled:
|
||
|
||
```bash
|
||
--dtype bfloat16 \
|
||
--mm-encoder-attn-dtype fp8
|
||
```
|
||
|
||
The first attempt on `cuda:0` failed before serving traffic because GPU 0 did not have enough free memory for the default `--gpu-memory-utilization 0.85` target.
|
||
|
||
The second attempt on `cuda:1` booted farther, but vLLM aborted during model initialization with:
|
||
|
||
```text
|
||
ValueError: mm_encoder_attn_dtype='fp8' requires the FlashInfer cuDNN backend with cuDNN >= 9.17.1 on a GPU with native FP8 support.
|
||
```
|
||
|
||
That means:
|
||
|
||
- no FP8 latency number was produced
|
||
- the current A100 hardware is not a valid target for this exact FP8 path
|
||
- the failure is architectural, not a tuning issue
|
||
|
||
To measure a real FP8 speed change, the next engineer should use a GPU with native FP8 support and a vLLM build that satisfies the FlashInfer cuDNN requirement for multimodal encoder FP8.
|
||
|
||
## NVIDIA Docker Fix
|
||
|
||
Docker currently has no registered `nvidia` runtime. The expected daemon config is:
|
||
|
||
```json
|
||
{
|
||
"data-root": "/data/docker_cache",
|
||
"runtimes": {
|
||
"nvidia": {
|
||
"args": [],
|
||
"path": "nvidia-container-runtime"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
This user cannot write `/etc/docker/daemon.json` or restart Docker because `sudo` requires a password. The host does support Docker GPU passthrough through modern `--gpus all`, so the API now patches Surya's vLLM spawn path to use `scripts/docker_nvidia_runtime_compat.sh`. That wrapper strips the legacy `--runtime nvidia` pair and preserves `--gpus device=...`.
|
||
|
||
An admin can still register the named runtime permanently with:
|
||
|
||
```bash
|
||
sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json
|
||
sudo systemctl restart docker
|
||
docker run --rm --runtime nvidia --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
|
||
```
|
||
|
||
The repo includes `scripts/configure_nvidia_docker.sh` with those commands.
|