Qwen3.5 for document extraction on vLLM: classify, don't generate
Layout models such as ERNIE-Layout and LayoutLMv3 are small and fast. On an A100, ERNIE-Layout labels an invoice page in about 7.5 ms. Vision-language models such as Qwen3.5 read documents much better, but they are typically used generatively: you ask for the fields and they write JSON. I wanted to know how close Qwen3.5 can get to ERNIE-Layout's speed for key-information extraction, and what it takes.
All experiments use the public RealKIE FCC invoices (CC BY-NC 4.0): 11 fields including 5 line-item fields, 368 test pages, scored with seqeval entity F1 over every word. Speeds are on one A100 shared with other jobs, with requests batched by vLLM.
Generation is the bottleneck
Zero-shot, asking for JSON:
| model | ms / page | output tokens / page | F1 |
|---|---|---|---|
| ERNIE-Layout (fine-tuned) | 7.5 | none | 0.667 |
| Qwen3-VL-2B, JSON | 183 | 792 | 0.23 |
| Qwen3.5-4B, JSON | 351 | 737 | 0.37 |
| Qwen3.5-4B, prefill only (no output) | 47 | 1 | n/a |
| Qwen3.5-9B FP8, prefill only | 107 | 1 | n/a |
Reading the page (prefill) takes 47 ms for the 4B model. Writing about 740 tokens of JSON takes the other 300 ms. The usual decode tricks helped less than I expected at batch 64: Qwen3.5's own MTP speculative decoding made the 4B model 7% faster and the 9B FP8 model slower, because a full batch is already compute-bound. N-gram speculation did halve the time when the OCR text was in the prompt, since answers copy that text. A more compact output format saved tokens but lost accuracy. Processing one page at a time instead of a batch took 2.3 s per page.
Classify instead
ERNIE-Layout is fast because it never generates: one forward pass tags every word. Qwen3.5 can do the same. Each page goes in as the image followed by its OCR words, and a linear head on the first sub-token of every word predicts the BIO tag. I trained LoRA adapters on the language model and the head (vision tower frozen), 2,000 steps, the same budget as the ERNIE-Layout baseline. The 0.8B model trains in 30 minutes on 3 GB of GPU memory, the 4B model in 70 minutes on 12 GB.
One wrinkle: Qwen3.5 is causal, so a word only sees the words before it (and the whole image). Feeding the words twice and reading the labels from the second copy ("echo") lets every word see the whole page's text, at the cost of twice the text tokens.
Results
| model | seeds | F1 | vs ERNIE-Layout | ms / page (vLLM) |
|---|---|---|---|---|
| Qwen3.5-4B, echo | 3 | 68.98 ± 0.85 | +2.24 (p = 0.02) | 148 |
| Qwen3.5-0.8B, no echo | 5 | 67.43 ± 0.27 | +0.69 (p = 0.19, a tie) | 28 |
| LayoutLMv3, whole page (2048 tokens) | 5 | 67.56 | ||
| ERNIE-Layout, whole page (2048 tokens) | 5 | 66.99 | ||
| ERNIE-Layout, 512-token windows | 5 | 66.74 | 7.5 | |
| LayoutLMv3, 512-token windows | 5 | 66.39 | 12.1 |
The 4B classifier is significantly better than both ERNIE-Layout settings, by about 2 points. The 0.8B classifier ties them at 28 ms per page, 3.7 times ERNIE-Layout's time. Compared with generating JSON, classification is more than 10 times faster and nearly doubles F1. (A single seed of the 0.8B model first looked better than ERNIE-Layout; five seeds say it is a tie.)
Where the remaining time goes
A torch profile of the 0.8B model in vLLM splits GPU time about 24% vision encoder and 76% language model. Without the image, the same model takes 10 ms per page, but F1 drops to 0.57: the image matters. The cheaper knobs trade accuracy for speed:
- Dropping echo: 40.8 to 28 ms, -0.6 F1. Worth it.
- Halving image resolution (about 390 image tokens): 28 to 20 ms, -1.2 F1.
- Quartering it (about 200 image tokens, measured with echo): 40.8 to 30.4 ms, -2.5 F1.
- vLLM's opt-in encoder CUDA graphs and FlashInfer encoder autotuning: no change. At about 3,000 patches per page the encoder is compute-bound, not launch-bound.
What is left is set by parameters times tokens. ERNIE-Layout has about 0.3B parameters and reads 512-token windows; Qwen3.5-0.8B reads about 2,000 tokens per page. Matching ERNIE-Layout exactly would need fewer tokens or cheaper arithmetic (FP8 on GPUs that support it), not a faster serving engine.
Serving it with vLLM
vLLM can already run any supported model as a token classifier: convert="classify" adds a score head and the pooling runner returns per-token probabilities. After merging the LoRA weights, I saved the head as score.weight and score.bias next to the model weights. vLLM's F1 matched the training code (0.6785 vs 0.6790 for 0.8B, 0.6979 vs 0.6976 for 4B).
from vllm import LLM
llm = LLM(model="path/to/merged-qwen3.5-classifier",
runner="pooling", convert="classify",
pooler_config={"task": "token_classify"},
limit_mm_per_prompt={"image": 1}, max_num_batched_tokens=16384)
# prompt_token_ids: vision_start, ONE <|image_pad|>, vision_end, then the word pieces
out = llm.encode([{"prompt_token_ids": ids, "multi_modal_data": {"image": page}}],
pooling_task="token_classify")
probs = out[0].outputs.data # (num_tokens, num_labels)
Two traps cost me time:
- Set the pooler task. Without
pooler_config={"task": "token_classify"}, the entrypoint picksclassify(one label per sequence) andencode(..., pooling_task="token_classify")is rejected. - Pass a single image placeholder. If
prompt_token_idsalready contain the expanded image pads that the Hugging Face processor produces, vLLM expands them again: a 3,663-token page became 4,406 tokens and every label shifted, with no error (F1 0.02). Maintainers consider correct placeholders the client's job (see #57740).
Cheaper GPUs
The 0.8B classifier needs about 3 GB at inference, so it fits a 3090 or 4090 easily. Older cards take more work. Current vLLM releases ship CUDA 13 wheels only, and vLLM dropped Volta (V100) from its official builds in April 2026. To run current main on CUDA 12, I built it from source against torch==2.14.1+cu126 (which still includes sm70) with GCC 13, and removed the CUDA-13-only optional dependencies. The build then failed at startup for every model because a MiniMax-M3 warmup module is imported unconditionally and needs one of them; that is #49920, with a fix in #49933. In fp16 (the only half precision a V100 has), Qwen3-VL-2B gave the same speed and accuracy as bf16 on these pages.
Contributed upstream
- vllm-project/vllm#68804 adds a DocVQA dataset to
vllm bench. No existing benchmark dataset had document pages, whose requests carry thousands of image tokens and short answers. - A reproduction on current
mainfor #49920, which was about to be closed as stale.
Caveats
This is one dataset of one document type, and every page is processed on its own. The A100 was shared, so treat times as relative. Accuracy comparisons use Welch's t-test over seeds; the 4B model has 3 seeds.
I used an AI coding assistant (Claude) to help write the code, run the experiments and draft this post, and I checked the results myself.