LayoutLMv3 now runs on SDPA in Transformers
Until this week, LayoutLMv3 in Hugging Face Transformers only had a hand-written eager attention, and every layer rebuilt its relative position biases from scratch. My PR huggingface/transformers#50688 moved it to the library's attention interface, so it now runs on PyTorch's SDPA and on flex attention. The same PR closed the long-standing request for it, #35467. The PR was merged on 17 November 2026, so the change is on main and will ship in the next release.
The short version, for token classification with microsoft/layoutlmv3-base on an A100 with fp16 autocast:
| text tokens | mode | batch | before (eager) | now (SDPA) |
|---|---|---|---|---|
| 512 | inference | 8 | 51.7 ms / 1.81 GiB | 28.3 ms / 1.27 GiB |
| 512 | train step | 8 | 129.7 ms / 9.58 GiB | 64.4 ms / 5.06 GiB |
| 2048 | inference | 2 | 119.2 ms / 3.45 GiB | 49.2 ms / 2.07 GiB |
| 2048 | train step | 2 | out of memory (15 GiB cap) | 94.3 ms / 6.98 GiB |
Times are the mean of 10 iterations, peak memory is in GiB, and every other sample in the batch is padded. The GPU was shared with other jobs, so treat the times as rough. The 2048-token rows use text position embeddings resized to 2050. That is not a published configuration, but it is how I run LayoutLMv3 on whole pages (see Long documents with LayoutLMv3 on a 16 GB GPU).
What was slow
LayoutLMv3 adds two learned relative position biases to the attention scores. One is over token positions. The other is over the x and y coordinates of the bounding boxes. Both have shape (batch, heads, n, n), and both are identical in every layer. The old code still rebuilt them in each layer. It bucketed every (query, key) pair in fp32, gathered from the bias tables, and then ran its own softmax. The softmax was a numerically stabilized variant from the CogView paper. That made SDPA impossible and created several n x n fp32 temporaries per layer. At 2048 tokens, those temporaries are what ran a training step out of memory.
Three changes
1. Attention through the interface. The self-attention now dispatches through ALL_ATTENTION_FUNCTIONS, like BERT, and passes the bias as position_bias, like T5. SDPA and flex attention both accept an additive bias, so they work without special cases. FlashAttention does not accept one, so it stays unsupported. That is the same situation as UDOP.
2. Build the biases once. The encoder computes the combined bias once per forward pass and passes it to every layer.
3. Bucket the distances, not the pairs. A relative-position bucket depends only on the distance, and it saturates beyond max_distance. So the code buckets the 2 * max_distance + 1 possible distances once, clamps the pairwise distances, and gathers with one index_select. A new test checks this against bucketing every pair, with zero tolerance, on 600 positions and box coordinates up to 1000.
Why the outputs did not change
Swapping the softmax was the part I was most careful about, because LayoutLMv3 has many fine-tuned checkpoints downstream. The CogView trick computes (x/α - max(x/α)) · α before the softmax, with α = 32. Because 32 is a power of two, dividing and multiplying by it is exact in floating point. The expression is therefore exactly x - max(x), and a softmax subtracts the max internally anyway. I checked this on the real model instead of trusting the argument. With microsoft/layoutlmv3-base, a batch of 3 with padding and 512 text tokens plus 197 image patches, the eager fp32 outputs of the new code are bit-identical to the old code on CPU and on an A100. The 211 gradients differ by at most 1.1e-7, because the old max subtraction sat in the backward graph. Against eager, SDPA differs by 3.2e-6 on CPU and 1.6e-5 on the A100 (token classification logits, fp32). In fp16 and bf16 the old and new code differ at rounding level.
What review changed
vasqu reviewed the PR for Hugging Face. Three of his requests changed the design:
- Pass the bias through the interface. My first version merged the bias into the attention mask once and shared the mask across layers. He asked for the standard
position_biasargument instead. It is cleaner, but in training SDPA now combines the bias with the padding mask in every layer and keeps each copy for backward. The 2048-token training step peaks at 6.98 GiB instead of 3.54 GiB. I flagged the trade-off in the PR. It is still far below the old code, which ran out of memory. - Use the standard eager function. This one had a hidden reason. PP-DocLayoutV2 reuses LayoutLMv3's classes through Transformers' "modular" code generation, and its generated file has a single
eager_attention_forwardthat its RT-DETR attention also calls. A CogView version under that name would have changed the detector's attention too. With the plain softmax, PP-DocLayoutV2's outputs stay bit-identical. Its reading-order model now supports SDPA and flex as well: in fp32, eager, SDPA and flex agree within 4.8e-8. - Fix the test setup, not the test. The common flex test enlarges
hidden_sizewhen the head size is below 16, which breaks LayoutLMv3's layout embeddings. I had overridden the test. He suggested changing the tester's initial values instead. With 2 heads instead of 4, the head size is 18 and the common test runs unchanged.
Using it
Nothing changes in your code: SDPA is the default when it is available. The snippet below runs on current main:
import torch
from PIL import Image
from transformers import AutoProcessor, LayoutLMv3ForTokenClassification
processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False)
model = LayoutLMv3ForTokenClassification.from_pretrained("microsoft/layoutlmv3-base", num_labels=5)
print(model.config._attn_implementation) # "sdpa"
image = Image.new("RGB", (800, 1000), "white")
words = ["Invoice", "number", "12345"]
boxes = [[10, 10, 80, 30], [90, 10, 160, 30], [170, 10, 230, 30]]
inputs = processor(image, words, boxes=boxes, return_tensors="pt")
with torch.no_grad():
print(model(**inputs).logits.shape) # torch.Size([1, 7, 5])
Two things to know:
output_attentions=Trueneedsattn_implementation="eager", as for BERT.- The PR title carries Transformers' 🚨 marker for breaking changes. The private attention helpers changed, so code that patched
cogview_attentionor the per-layerrel_pos/rel_2d_posarguments needs updating.
I used an AI coding assistant (Claude) to help write the code, tests and benchmarks, and I checked the results myself.