Qiming Bao · Blog · 17 November 2026 · English / 中文

LayoutLMv3 now runs on SDPA in Transformers

Until this week, LayoutLMv3 in Hugging Face Transformers only had a hand-written eager attention, and every layer rebuilt its relative position biases from scratch. My PR huggingface/transformers#50688 moved it to the library's attention interface, so it now runs on PyTorch's SDPA and on flex attention. The same PR closed the long-standing request for it, #35467. The PR was merged on 17 November 2026, so the change is on main and will ship in the next release.

The short version, for token classification with microsoft/layoutlmv3-base on an A100 with fp16 autocast:

text tokensmodebatchbefore (eager)now (SDPA)
512inference851.7 ms / 1.81 GiB28.3 ms / 1.27 GiB
512train step8129.7 ms / 9.58 GiB64.4 ms / 5.06 GiB
2048inference2119.2 ms / 3.45 GiB49.2 ms / 2.07 GiB
2048train step2out of memory (15 GiB cap)94.3 ms / 6.98 GiB

Times are the mean of 10 iterations, peak memory is in GiB, and every other sample in the batch is padded. The GPU was shared with other jobs, so treat the times as rough. The 2048-token rows use text position embeddings resized to 2050. That is not a published configuration, but it is how I run LayoutLMv3 on whole pages (see Long documents with LayoutLMv3 on a 16 GB GPU).

What was slow

LayoutLMv3 adds two learned relative position biases to the attention scores. One is over token positions. The other is over the x and y coordinates of the bounding boxes. Both have shape (batch, heads, n, n), and both are identical in every layer. The old code still rebuilt them in each layer. It bucketed every (query, key) pair in fp32, gathered from the bias tables, and then ran its own softmax. The softmax was a numerically stabilized variant from the CogView paper. That made SDPA impossible and created several n x n fp32 temporaries per layer. At 2048 tokens, those temporaries are what ran a training step out of memory.

Three changes

1. Attention through the interface. The self-attention now dispatches through ALL_ATTENTION_FUNCTIONS, like BERT, and passes the bias as position_bias, like T5. SDPA and flex attention both accept an additive bias, so they work without special cases. FlashAttention does not accept one, so it stays unsupported. That is the same situation as UDOP.

2. Build the biases once. The encoder computes the combined bias once per forward pass and passes it to every layer.

3. Bucket the distances, not the pairs. A relative-position bucket depends only on the distance, and it saturates beyond max_distance. So the code buckets the 2 * max_distance + 1 possible distances once, clamps the pairwise distances, and gathers with one index_select. A new test checks this against bucketing every pair, with zero tolerance, on 600 positions and box coordinates up to 1000.

Why the outputs did not change

Swapping the softmax was the part I was most careful about, because LayoutLMv3 has many fine-tuned checkpoints downstream. The CogView trick computes (x/α - max(x/α)) · α before the softmax, with α = 32. Because 32 is a power of two, dividing and multiplying by it is exact in floating point. The expression is therefore exactly x - max(x), and a softmax subtracts the max internally anyway. I checked this on the real model instead of trusting the argument. With microsoft/layoutlmv3-base, a batch of 3 with padding and 512 text tokens plus 197 image patches, the eager fp32 outputs of the new code are bit-identical to the old code on CPU and on an A100. The 211 gradients differ by at most 1.1e-7, because the old max subtraction sat in the backward graph. Against eager, SDPA differs by 3.2e-6 on CPU and 1.6e-5 on the A100 (token classification logits, fp32). In fp16 and bf16 the old and new code differ at rounding level.

What review changed

vasqu reviewed the PR for Hugging Face. Three of his requests changed the design:

Using it

Nothing changes in your code: SDPA is the default when it is available. The snippet below runs on current main:

import torch
from PIL import Image
from transformers import AutoProcessor, LayoutLMv3ForTokenClassification

processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False)
model = LayoutLMv3ForTokenClassification.from_pretrained("microsoft/layoutlmv3-base", num_labels=5)
print(model.config._attn_implementation)  # "sdpa"

image = Image.new("RGB", (800, 1000), "white")
words = ["Invoice", "number", "12345"]
boxes = [[10, 10, 80, 30], [90, 10, 160, 30], [170, 10, 230, 30]]
inputs = processor(image, words, boxes=boxes, return_tensors="pt")
with torch.no_grad():
    print(model(**inputs).logits.shape)  # torch.Size([1, 7, 5])

Two things to know:

I used an AI coding assistant (Claude) to help write the code, tests and benchmarks, and I checked the results myself.