Suya OCR API — vLLM-backed, OpenAI-compatible OCR service
FastAPI service wrapping the Surya-OCR-2 model (datalab-to) served through vLLM: legacy /v1/api/ai/* endpoints, an OpenAI-compatible /v1/chat/completions endpoint, a coalescing request batcher, a local OCR CLI, Docker packaging, multilingual example outputs, and quantization/concurrency benchmarks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Fu Dai
@@ -0,0 +1,182 @@
|
||||
# vLLM Backend Plan and Test Results
|
||||
|
||||
Date: 2026-06-09
|
||||
|
||||
## Plan
|
||||
|
||||
1. Keep the legacy OCR endpoint at `/v1/api/ai/suya_ocr/` for baseline comparison.
|
||||
2. Add `/v1/api/ai/suya_ocr_vllm/`, backed by Surya2 `SuryaInferenceManager(method="vllm")`.
|
||||
3. Use the Surya vLLM backend settings from the referenced Surya checkout:
|
||||
- `SURYA_INFERENCE_BACKEND=vllm`
|
||||
- `SURYA_INFERENCE_KEEP_ALIVE=true`
|
||||
- T4 default: `VLLM_GPU_TYPE=t4`, `VLLM_DTYPE=float16`, `SURYA_INFERENCE_PARALLEL=16`
|
||||
- optional tuning through `VLLM_GPUS`, `VLLM_GPU_MEMORY_UTILIZATION`, and `SURYA_INFERENCE_URL`
|
||||
4. Add a concurrency benchmark that sends the same payload to the legacy and vLLM endpoints and writes latency/throughput JSON.
|
||||
5. Verify syntax locally, then run the benchmark on a host with the Surya2 package, Docker, and NVIDIA GPU access.
|
||||
|
||||
## Implemented Files
|
||||
|
||||
- `vllm_tools.py`: vLLM-backed Surya2 predictor setup and OCR response formatting.
|
||||
- `api.py`: new `/v1/api/ai/suya_ocr_vllm/` endpoint and health endpoint.
|
||||
- `concurrency_test.py`: repeatable old-vs-new endpoint concurrency benchmark.
|
||||
- `concurrency_test_results.json`: placeholder result file that the benchmark overwrites with measured results.
|
||||
- `requirements.txt`: dependency alignment for Surya2/vLLM OpenAI client support.
|
||||
- `Dockerfile`: T4 vLLM defaults. The referenced backend maps T4 16 GB VRAM to `max_num_seqs=16` and `max_num_batched_tokens=4096`; `float16` is required because T4 does not support `bfloat16`.
|
||||
|
||||
## Single-Container Kubernetes Deployment
|
||||
|
||||
The deployment now uses `vllm/vllm-openai:v0.20.1` as the base image. The container starts the vLLM OpenAI-compatible server in the background and runs `api.py` in the foreground through `scripts/start_single_container.sh`.
|
||||
|
||||
The API no longer depends on nested Docker. It attaches to:
|
||||
|
||||
```text
|
||||
SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1
|
||||
```
|
||||
|
||||
The Surya source needed by the API is vendored into this repo under `surya/`, so the container does not depend on `/path/to/surya` at runtime.
|
||||
|
||||
Serving defaults for T4:
|
||||
|
||||
```text
|
||||
SUYA_OCR_MODE=block
|
||||
SUYA_MAX_BATCH_SIZE=8
|
||||
SUYA_BATCH_WAIT_MS=25
|
||||
SURYA_INFERENCE_PARALLEL=8
|
||||
SURYA_INFERENCE_MAX_RETRIES=1
|
||||
SURYA_MAX_BLOCKS_PER_PAGE=80
|
||||
VLLM_DTYPE=float16
|
||||
VLLM_MAX_NUM_SEQS=16
|
||||
VLLM_MAX_BATCHED_TOKENS=4096
|
||||
```
|
||||
|
||||
Speed-related changes:
|
||||
|
||||
- `/v1/api/ai/suya_ocr_vllm/` uses `vllm_batcher.py` to coalesce concurrent HTTP requests into batch Surya calls.
|
||||
- `vllm_tools.py` batches layout and recognition across images.
|
||||
- Default OCR mode is block mode, avoiding long full-page generations and hidden fallback bursts.
|
||||
- Vendored Surya caps block OCR per page with `SURYA_MAX_BLOCKS_PER_PAGE`.
|
||||
- Retry amplification is limited with `SURYA_INFERENCE_MAX_RETRIES=1`.
|
||||
- The endpoint skips drawing annotated images because the API response only returns JSON/text.
|
||||
- Base64 input is decoded once per request and the same PIL image object is reused for low/high-res OCR paths unless a caller explicitly asks for a copy.
|
||||
- vLLM image payload encoding defaults to high-quality JPEG (`SUYA_VLLM_IMAGE_FORMAT=JPEG`, `SUYA_VLLM_JPEG_QUALITY=92`) to reduce CPU serialization and local vLLM transfer size versus PNG. Set `SUYA_VLLM_IMAGE_FORMAT=PNG` if lossless transport is required.
|
||||
|
||||
## Benchmark Command
|
||||
|
||||
```bash
|
||||
python concurrency_test.py \
|
||||
--image /path/to/surya/ttt.png \
|
||||
--requests 20 \
|
||||
--concurrency 8 \
|
||||
--output concurrency_test_results.json
|
||||
```
|
||||
|
||||
## Current Test Results
|
||||
|
||||
Import verification passed with `PYTHONPATH=/path/to/surya`.
|
||||
|
||||
Concurrency benchmark run:
|
||||
|
||||
```bash
|
||||
/opt/conda/bin/python concurrency_test.py \
|
||||
--image temp_image_c3c56948-282e-453b-8fac-6c482243d1e5.jpg \
|
||||
--requests 16 \
|
||||
--concurrency 16 \
|
||||
--timeout 900 \
|
||||
--output concurrency_test_results.json
|
||||
```
|
||||
|
||||
Run started at `2026-06-11T14:35:17+0400`. Docker NVIDIA runtime and GPU checks passed. The local Python environment does not include the `vllm` package, so this validation attached the updated API to the already-warm local `vllm/vllm-openai:v0.20.1` server on port `43401` with `SURYA_INFERENCE_URL=http://127.0.0.1:43401/v1`. The API path still used the single-container attach mode (`SURYA_INFERENCE_AUTOSTART=false`) and did not start nested Docker.
|
||||
|
||||
| Endpoint | Requests | Concurrency | Success | Failed | Wall seconds | Throughput rps | Mean latency | P50 latency | P95 latency | Max latency |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| `/v1/api/ai/suya_ocr/` | 16 | 16 | 16 | 0 | 43.62 | 0.3668 | 40.81 | 41.84 | 43.39 | 43.62 |
|
||||
| `/v1/api/ai/suya_ocr_vllm/` | 16 | 16 | 16 | 0 | 83.59 | 0.1914 | 59.08 | 60.41 | 83.52 | 83.54 |
|
||||
|
||||
Latency values are per request in seconds. Mean latency is the average request elapsed time, not the total benchmark wall time.
|
||||
|
||||
## Approach A Outcome (2026-06-12)
|
||||
|
||||
Diagnosis + work-reduction tuning (see `docs/diagnosis_baseline.md` and
|
||||
`docs/tier2_results.md`). Measured on the A100 dev host against the warm
|
||||
`surya-vllm-43401` server — same host/server as the table above, so comparable.
|
||||
**Production is T4; the fix below is hardware-independent, the absolute numbers
|
||||
are not.**
|
||||
|
||||
**Root cause:** the request batcher coalesced requests then dispatched them only
|
||||
`SURYA_INFERENCE_PARALLEL=8` wide, so only 8 of vLLM's 16 sequence slots were
|
||||
used (poller `peak_running=8`). The GPU sat half-idle, which is why the vLLM
|
||||
endpoint was *slower* than the flooding legacy path.
|
||||
|
||||
**Fix (Tier 1):** scale the worker pool to the in-flight block count, capped by a
|
||||
new `SURYA_INFERENCE_MAX_INFLIGHT` (default 16, = `VLLM_MAX_NUM_SEQS`). This fills
|
||||
all 16 slots (`peak_running=16`).
|
||||
|
||||
| vLLM endpoint, 16/16 | mean | p50 | p95 | peak_running |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| MI=8 (before) | 50.22 | 49.48 | 72.69 | 8 |
|
||||
| MI=16 (fix) | 33.75 | 36.11 | 49.15 | 16 |
|
||||
|
||||
- Mean −33%, p95 −32%; single request 6.16 s → ~5.3 s. CER divergence 0.0009
|
||||
(lossless; gate ≤ 0.005). The vLLM endpoint now beats legacy on mean and median.
|
||||
- MI=24 overcommits (worse). Batch-size / batch-wait are within noise.
|
||||
|
||||
**Tier 2 (token ceiling, recognition DPI, block cap):** no latency benefit on
|
||||
this sparse page — decode dominates and the page is light — so all Tier-2 knobs
|
||||
are left at defaults. They remain relevant for dense / pathological pages.
|
||||
|
||||
**Locked change:** `SURYA_INFERENCE_MAX_INFLIGHT=16` added to `Dockerfile` and
|
||||
`scripts/start_single_container.sh`. Re-confirm absolute latencies on a T4 before
|
||||
treating them as production SLAs; the regression fix itself applies unchanged.
|
||||
|
||||
## FP8 Experiment
|
||||
|
||||
I attempted to start the latest vLLM image with multimodal encoder FP8 enabled:
|
||||
|
||||
```bash
|
||||
--dtype bfloat16 \
|
||||
--mm-encoder-attn-dtype fp8
|
||||
```
|
||||
|
||||
The first attempt on `cuda:0` failed before serving traffic because GPU 0 did not have enough free memory for the default `--gpu-memory-utilization 0.85` target.
|
||||
|
||||
The second attempt on `cuda:1` booted farther, but vLLM aborted during model initialization with:
|
||||
|
||||
```text
|
||||
ValueError: mm_encoder_attn_dtype='fp8' requires the FlashInfer cuDNN backend with cuDNN >= 9.17.1 on a GPU with native FP8 support.
|
||||
```
|
||||
|
||||
That means:
|
||||
|
||||
- no FP8 latency number was produced
|
||||
- the current A100 hardware is not a valid target for this exact FP8 path
|
||||
- the failure is architectural, not a tuning issue
|
||||
|
||||
To measure a real FP8 speed change, the next engineer should use a GPU with native FP8 support and a vLLM build that satisfies the FlashInfer cuDNN requirement for multimodal encoder FP8.
|
||||
|
||||
## NVIDIA Docker Fix
|
||||
|
||||
Docker currently has no registered `nvidia` runtime. The expected daemon config is:
|
||||
|
||||
```json
|
||||
{
|
||||
"data-root": "/data/docker_cache",
|
||||
"runtimes": {
|
||||
"nvidia": {
|
||||
"args": [],
|
||||
"path": "nvidia-container-runtime"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This user cannot write `/etc/docker/daemon.json` or restart Docker because `sudo` requires a password. The host does support Docker GPU passthrough through modern `--gpus all`, so the API now patches Surya's vLLM spawn path to use `scripts/docker_nvidia_runtime_compat.sh`. That wrapper strips the legacy `--runtime nvidia` pair and preserves `--gpus device=...`.
|
||||
|
||||
An admin can still register the named runtime permanently with:
|
||||
|
||||
```bash
|
||||
sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json
|
||||
sudo systemctl restart docker
|
||||
docker run --rm --runtime nvidia --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
The repo includes `scripts/configure_nvidia_docker.sh` with those commands.
|
||||
Reference in New Issue
Block a user