Qiming Bao · Blog · 12 November 2026 · English / 中文

Re-implementing LayoutLMv3 pre-training: what continued pre-training did and did not buy

LayoutLMv3 is pre-trained with three objectives: masked language modelling (MLM), masked image modelling (MIM) and word-patch alignment (WPA). Microsoft released the fine-tuning code and the checkpoints, but not the pre-training code, and people keep asking for it (unilm#733, #1042, #1609 and others). I wrote a small, tested re-implementation on top of Hugging Face transformers: 14H034160212/layoutlmv3-pretraining. It is an independent implementation from the paper, not Microsoft's code.

This post is about what I learned from using it for continued pre-training of microsoft/layoutlmv3-base and -large. The short version: the released encoder is a strong starting point, my code builds on it correctly, and adding up to 400k more document pages gave at most a small gain on FUNSD and none on CORD.

What is implemented

One difference is deliberate. The paper's MIM targets come from DiT's document dVAE, which was never released, so I use the public DALL-E dVAE, which was trained on natural images. The output is a plain LayoutLMv3Model checkpoint that the usual task heads load directly.

Lesson 1: check where continued pre-training really starts

My first run looked like training from zero: validation loss 24.8 and MLM accuracy 0.00 at step 0. The reason is that the released checkpoints contain only the encoder. The MLM, MIM and WPA heads and the image [MASK] embedding are random at step 0, so step-0 metrics measure random heads, not the encoder. Training everything jointly from there also lets random-head gradients push on a pretrained encoder.

The fix is a head warm-up: 1,000 steps that train only the new parameters with the encoder frozen. I checked that the encoder is bit-identical afterwards, and fine-tuning it gives exactly the released model's FUNSD score. The warm-up turns into a useful correctness check:

A new head can only read masked tokens from a frozen encoder that well if the checkpoint is loaded correctly and the input pipeline (tokenisation, 0-1000 segment boxes, span masking, image normalisation) matches what the encoder saw in pre-training. It does not prove the code is equivalent to Microsoft's. Still, it is a much better check than a loss curve that starts at 24.8. It also exposed something about my first run: its final MLM accuracy after 6,000 joint steps was 0.515, barely above the 0.511 the frozen encoder gives after the warm-up. Most of that curve was the head being learned.

Lesson 2: decide what counts as a gain before you look

The paired difference in fine-tuning F1 between two checkpoints varies by about 0.75 points (standard deviation) from seed to seed, which is as large as any effect I was looking for. So before each experiment I committed the plan and the decision rule to the repository. For the scale-up the rule was: over 5 seeds × 2 tasks, the mean paired gain must be at least +0.5 F1 points, positive on both FUNSD and CORD, with p < 0.05.

Ten times more pages gave a clearly lower pre-training validation loss (8.38 after the warm-up, 6.21 at the best step) and higher MLM and MIM accuracy, but did not meet the rule downstream. With 10 seeds and a stricter evaluation, the 402k checkpoint gains +0.65 F1 on FUNSD (p = 0.047, not corrected for multiple comparisons) and nothing on CORD. That fits the data: the Industry Documents Library is mostly forms and letters, like FUNSD, while CORD is receipts. The released model was already pre-trained on about 11M pages of similar documents, so 400k more general pages is a small addition. In-domain pages for the target task look like the more promising direction.

Lesson 3: some objectives are easier to fool than they look

WPA reaches 99% accuracy quickly, and that number means little. Whether a token's patches are masked can be read off the learned [MASK] patch embedding. A head that also looks at the token's patches makes the shortcut obvious: with a random encoder it reaches AUC 1.000. I kept that variant in the code only so the shortcut can be reproduced, and report WPA as balanced accuracy and AUC on a text-only head. There, the released encoder gives AUC 0.904 with a two-layer MLP against at most 0.69 for a random one, so it does carry alignment information.

An independent code review also found three data-pipeline bugs: about 10% of pages were rotated against their OCR boxes, all 12 DataLoader workers drew identical masks, and resuming replayed the start of the corpus. Each now has a regression test. Re-running the 400k experiment with the fixes did not measurably change the downstream results (p = 0.35), but I would not have known that without re-running it.

If you want to use it

pip install -e ".[dev]"
pytest tests
python scripts/pretrain.py --corpus idl --idl_shards 0-26 --exclude_funsd_test \
    --image_tokenizer dalle --head_warmup_steps 1000 --output_dir runs/idl_base

Every run is recorded with its exact command, git commit, configuration and metrics, and new runs can log to Weights & Biases. Note the licences: the code is MIT, but the microsoft/layoutlmv3-* weights are CC BY-NC-SA 4.0, and checkpoints continued from them inherit that. The repository has a CITATION.cff; if you use the code, please cite the repository and the original LayoutLMv3 paper.

I used Claude (Anthropic) to help write the code and draft this post. I reviewed the design and the results, and every number here comes from the scripts and result files in the repository.

Qiming Bao is an AI researcher in Auckland, New Zealand, working on LLM reasoning and model deployment. He received his PhD from the University of Auckland's Strong AI Lab and contributes to the PyTorch ONNX exporter, ONNX Runtime, ONNX Runtime GenAI and Microsoft PyRIT.