Qiming Bao · Blog · 13 November 2026 · English / 中文

Long documents with LayoutLMv3 on a 16 GB GPU, without pre-training

LayoutLMv3 reads at most 512 text tokens. That is fine for a receipt, but a page of a statement or an order form with dozens of line items is often 1,000 tokens or more, and everything after token 512 is never seen. I wanted a version that reads the whole page under three constraints: no extra pre-training, inference on a 16 GB V100, and memory and speed close to the 512-token model. This post describes what worked. The code is in my layoutlmv3-pretraining repository.

How much does truncation lose?

FUNSD, the usual benchmark, is too short to tell: only 4 of its 50 test forms exceed 512 tokens. I used the Ad-buy Forms part of VRDU (Google's visually rich document benchmark): TV advertising orders whose pages are tables of line items, with 14 annotated fields such as channel, program, dates and amounts. I turned each page into token classification (BIO tags over the fields).

At 512 tokens the model cannot see four in ten of the words it is asked to label.

Reading longer pages without pre-training

Two things stop LayoutLMv3 from reading more than 512 tokens: it has only 512 learned position embeddings, and attention memory grows with the square of the length.

Positions. I extend the 1D position embeddings by repeating the 512 pretrained ones (the copy initialisation Longformer used), then fine-tune on the task as usual. Nothing else is trained. This works better than it sounds because LayoutLMv3 also has 2D layout embeddings and relative-position biases, which are unchanged by the length. The alternative is 512-token sliding windows with overlap, merging the predictions per word.

Results (LayoutLMv3-base, 2,000 fine-tuning steps, batch 2, 5 seeds, every word scored):

Reading the whole page at 2048 tokens is 24 points better than truncation and 1.6 points better than windows. The likely reason is that a window sees only part of a table, while the full page lets every row attend to the column headers and the other rows. 1024 tokens is not enough here: a third of the pages are longer, and their tails are still cut. Runs are logged on W&B (512, windows, 1024, 2048).

Convergence. Repeated position embeddings do not slow fine-tuning down. Evaluating every 250 steps (3 seeds), the training loss falls at the same rate with 512 and 2048 tokens, and the 2048-token model is ahead on the test set at every checkpoint: 36.3 against 19.0 F1 at step 250, 74.1 against 53.7 at step 500 (already above truncation's final 68.3), 89.5 against 66.3 at step 1000. It levels off near step 1500 at 92.7 (W&B 512, 2048). This run used the lookup-table attention for training as well.

ERNIE-Layout (through the community PyTorch port, with the same position tiling) shows the same picture on the same pages: truncation 70.87 ± 0.18, windows 92.78 ± 0.30, positions extended to 2048 92.88 ± 0.17 (W&B 512, windows, 2048). Reading the whole page is again 22 points better than truncation, but for ERNIE-Layout it ties with windows (+0.10, paired p = 0.28), while for LayoutLMv3 the 1.6-point gain over windows held in all five seeds (p = 0.007).

Speed comes from dynamic padding: each batch is padded to its longest document instead of to 2048, so short pages cost the same as with a 512-token model and only genuinely long pages cost more. Fine-tuning at 2048 took 1.5 times as long as truncation, not 4 times.

Where the memory actually goes

LayoutLMv3 adds learned biases to every attention score, based on the 1D distance between two tokens and their horizontal and vertical distance on the page. In transformers, each bias is computed by building n × n matrices of distances, sorting them into logarithmic buckets in fp32, and gathering from the bias tables. I expected the n × n bias itself to be the problem. It is not: in fp16 it is about 121 MB at 2048 tokens plus 197 image patches. The fp32 temporaries of the bucketing are several times larger.

The bucket depends only on the distance, and there are only a few thousand possible distances. So I compute each table once per distance, gather the combined bias directly in fp16, and share it across all 12 layers. The result is exact: the same outputs as transformers' attention. Attention then runs through PyTorch's memory-efficient scaled_dot_product_attention kernel.

Inference, one document, fp16, total memory (weights included) / time:

To check that the lookup tables change nothing on real data, I took one model fine-tuned on VRDU at 2048 tokens and evaluated it on the 774 test pages with four attention implementations: the previous SDPA version, the lookup tables, SDPA over query blocks, and FlexAttention. All four gave F1 92.96 and the same prediction for every labelled token. In that evaluation (fp32, batch 8, padded to 2048) the lookup-table version was also the fastest and needed 7.5 GB against 14.8 GB (W&B).

At 2048 tokens the model needs 0.80 GB, against 0.69 GB at 512, and it is faster than the previous SDPA version. For fine-tuning at 2048 tokens with batch 2 it needs 7.05 GB, or 3.99 GB with gradient checkpointing; 4096 tokens fit in 6.71 GB with checkpointing.

FlexAttention, and why I don't recommend it for a V100

My first attempt was FlexAttention, which computes the bias inside the attention kernel and never builds an n × n tensor. On an A100 it is impressive: 0.74 GB at 2048 tokens and 0.87 GB at 8192. Its first version was about four times slower than SDPA at 2048 tokens (0.35 s against 0.09 s), because it bucketed every score; the per-distance tables fixed that, which is where the idea above came from.

But FlexAttention compiles its kernels with Triton, and recent Triton versions no longer use the tensor cores of pre-Ampere GPUs such as the V100. The A100 speed should not be expected there. The lookup-table version uses PyTorch's own memory-efficient kernel, which does use V100 tensor cores and needs no compiler. For pages beyond 8192 tokens there is also an option that runs SDPA over blocks of 512 queries: 1.08 GB at 8192 tokens, but 2.7 to 4.4 times slower than the lookup-table version (at 2048 to 8192 tokens).

A caveat for all of the numbers: I measured on an A100 with a 15 GB memory cap and only the attention kernels a V100 supports, not on a real V100.

What is new and what is not

Most of the ingredients are known. Longformer (2020) showed that copying position embeddings and fine-tuning without extra pre-training costs only about 0.6 points on WikiHop against its pre-trained version. LSG Attention (2022) converts BERT, RoBERTa and BART checkpoints to long inputs with sparse attention. LongEmbed (2024) extends embedding models without training. A recent paper (arXiv 2611.01182, October 2026) computes relative-position attention exactly inside FlashAttention-style kernels, written in Triton.

What I have not found elsewhere is the combination for layout models: LayoutLMv3's 1D and 2D biases, no pre-training, exact full attention (short pages give the same outputs as the original model), plain PyTorch kernels that run on a V100, and a benchmark with genuinely long pages. Sparse attention was needed in 2020 because full attention did not fit; with memory-efficient kernels, full attention fits at 2048 to 8192 tokens on a 16 GB GPU.

Next

• A second long-document dataset, and accuracy at 4096 tokens and beyond: VRDU has no labelled words past 2048.
• A comparison with LSG and Longformer on their long-text benchmarks, to test whether exact attention with copied positions holds up for plain-text encoders too.
• Factorising the layout biases into low-rank terms appended to the queries and keys, as FlashBias (NeurIPS 2025) does for other biases, so that no n × n bias is built at all.
• Confirming the numbers on a real V100.

Code, runs and upstream changes

• Repository: 14H034160212/layoutlmv3-pretraining. The README section "Long documents within a 16 GB GPU" has every table with its W&B run, and results/long_docs/ has the raw results.
• Position extension and the attention implementations: src/layoutlmv3_pretraining/long_docs.py. Fine-tuning: scripts/finetune_funsd.py. VRDU conversion: scripts/prepare_vrdu.py. Benchmarks: scripts/bench_inference_memory.py, scripts/bench_train_memory.py.
• The recommended setting on VRDU:
  python scripts/finetune_funsd.py --task vrdu --model microsoft/layoutlmv3-base --protocol strict --extend_positions --max_length 2048 --attention lut --dynamic_padding
• transformers: #50688 brings SDPA and the lookup-table bias to LayoutLMv3 (#35467).
• ERNIE-Layout port: #27 (image normalised twice), #30 (tokenizer on transformers 4.45 and newer), #31 (relative biases with index_select: 3.8 times faster training steps, and 4096-token inference fits).

I used Claude (Anthropic) to help implement the code, run the experiments and draft this post. I reviewed the results and the post myself.

Qiming Bao is an AI researcher in Auckland, New Zealand, working on LLM reasoning and model deployment. He received his PhD from the University of Auckland's Strong AI Lab and contributes to the PyTorch ONNX exporter, ONNX Runtime, ONNX Runtime GenAI and Microsoft PyRIT.