Qiming Bao · Blog · 13 November 2026 · English / 中文

Three silent bugs in fine-tuning LayoutLMv3 and ERNIE-Layout

While adding long-document support to my LayoutLMv3 pre-training repository, I also brought in ERNIE-Layout through the community PyTorch port. Along the way I found three bugs. None of them raises an error or a warning, and each one quietly makes a model worse than it should be. I reported all three, with fixes.

1. ERNIE-Layout's image is normalised twice

ERNIE-Layout's visual backbone is a ResNet that normalises its input itself:

images_input = (images - self.pixel_mean) / self.pixel_std
# pixel_mean = [103.53, 116.28, 123.675], pixel_std = [57.375, 57.12, 58.395]

Those constants expect pixels in 0-255. PaddleNLP, where the model comes from, feeds exactly that: the page decoded to RGB, resized to 224 x 224 and kept as 0-255 (_str2im followed by _permute(image, to_bgr=False)). The PyTorch port's README and examples, however, create the image processor as LayoutLMv3ImageProcessor(apply_ocr=False). Its defaults rescale to [0, 1] and normalise with mean = std = 0.5, so the pixels are already in [-1, 1] before the backbone normalises them again.

After the second normalisation every pixel sits near -1.97 with a standard deviation of 0.13: the model sees an almost black page. I measured what that does to the visual features on 8 FUNSD test pages:

With the README's setting, every page looks the same to the model. The fix is two arguments. One trap: the mean values look like the BGR-ordered Detectron constants, but PaddleNLP feeds RGB, and the model was pre-trained that way, so do not convert to BGR. Reported as NormXU/ERNIE-Layout-Pytorch#26 with the fix in #27.

If you already fine-tuned with the README's preprocessing, keep using it for those checkpoints, or fine-tune again. I fine-tuned ERNIE-Layout on FUNSD with each preprocessing (3 seeds) and evaluated every model with both. A model fine-tuned with the README's preprocessing scores 84.19 F1 with it, but 82.78 when switched to the corrected one (−1.4, paired p = 0.03). Fine-tuning again with the corrected preprocessing gives 84.55, slightly above the old setup. That difference is not significant with 3 seeds on FUNSD, which depends little on the image. Training and inference preprocessing have to change together (W&B: README preprocessing, corrected preprocessing).

LayoutLMv3 itself does not have this problem. Its model applies no normalisation of its own, and its processor's 0.5 / 0.5 matches Microsoft's original fine-tuning code. The bug only appears when ERNIE-Layout borrows LayoutLMv3's processor.

2. BatchNorm statistics change while the encoder is frozen

A common recipe for continued pre-training is a head warm-up: freeze the encoder and train only the new heads for a while. I check that the encoder is bit-identical afterwards. For ERNIE-Layout it was not.

The ResNet has 11 BatchNorm layers. In model.train() mode, BatchNorm updates its running mean and variance on every forward pass, whether or not its parameters require gradients. One forward pass with the encoder frozen changed the running statistics of all 11 layers. In fine-tuning with a batch of 2 pages, the pretrained statistics drift towards whatever those small batches contain.

The fix is the one detection backbones usually use: keep BatchNorm in eval mode during training. With that, a head warm-up changed 0 encoder parameters and 0 buffers. The general lesson is that requires_grad_(False) freezes parameters, not buffers.

3. transformers stopped training LayoutLMv2/v3's relative position biases in v4.42

LayoutLMv3 adds learned 1D and 2D relative position biases to its attention scores. In transformers they are looked up like this:

with torch.no_grad():
    rel_pos = self.rel_pos_bias.weight.t()[rel_pos].permute(0, 3, 1, 2)

The comment above it says the lookup "is independent of the input, no need to track gradients". The index does not depend on the input, but the result depends on the weight, so the three bias tables (rel_pos_bias, rel_pos_x_bias, rel_pos_y_bias) never receive a gradient. After a backward pass their .grad is None, while every other weight has one. They stay at their pretrained values in every fine-tuning or pre-training run.

The history explains how this happened. #26139 replaced a one-hot plus nn.Linear computation with indexing to save memory. Indexing has a very slow backward, and training became about 5x slower (#30974). #30988 then wrapped the lookup in no_grad instead of reverting, which fixed the speed and silently stopped the training of the tables. It has been in every release since v4.42.0, for LayoutLMv2 and LayoutLMv3.

The fix keeps the gradient without the slowdown. I measured a full training step of LayoutLMv3ForTokenClassification (base, 512 text tokens plus 197 patches, batch 2, A100):

index_select's backward accumulates with index_add_, so the tables train again for about 7% more time per step, and the forward result is unchanged. Reported as huggingface/transformers#50636, with the fix and a regression test in #50641, which will be reopened once the issue is triaged.

How to catch this kind of bug

• Compare inputs, not only outputs. The ERNIE conversion was checked against PaddleNLP by feeding both the same tensor, which can never catch a preprocessing difference. Check what the first layer actually receives.
• After freezing, check buffers as well as parameters. Snapshot named_buffers() before and after a frozen phase.
• After a backward pass, check that every trainable parameter has a gradient. A parameter with requires_grad=True and grad is None is almost always a bug.

All three checks are cheap enough to run as unit tests. The repository now has them.

I used Claude (Anthropic) to help trace these bugs, run the measurements and draft this post. I reviewed the results and ran the checks myself.

Qiming Bao is an AI researcher in Auckland, New Zealand, working on LLM reasoning and model deployment. He received his PhD from the University of Auckland's Strong AI Lab and contributes to the PyTorch ONNX exporter, ONNX Runtime, ONNX Runtime GenAI and Microsoft PyRIT.