Do layout models need the table header? A negative result
In Long documents with LayoutLMv3 on a 16 GB GPU, one of the baselines splits each page into 512-token windows. A window that starts in the middle of a line-item table cannot see the table's header. Neither can a continuation page where the header is not printed again. It seems obvious that this should hurt: how does the model know a column is the rate and not the number of spots, if it never sees the word "Rate"?
I tested it on RealKIE FCC invoices (TV ad invoices with long line-item tables) with LayoutLMv3 and ERNIE-Layout. The short answer: the models learned to find the header reliably, and giving them the header did not help. Without any header, they already get about 86 line-item F1 on headerless chunks.
Finding the header
The first question is how the model knows which rows are the header. I tried three things on 30 hand-checked test pages:
- Table Transformer (
microsoft/table-structure-recognition-v1.1-all), the standard off-the-shelf model: correct on 5 of 30 pages. These are low-resolution scans with borderless, multi-line headers, which is far from its training data. - Rules (the row above the first line item, or a word list mined from training data): 10 to 12 of 30.
- Learned: label the header rows on each training document's first line-item page, copy the labels to repeated headers on later pages, and train a
B-TABLE_HEADERtag together with the fields. This finds header words with precision 0.88 to 0.92 and recall 0.97, for both models.
The labels (202 training and 67 test documents) were made from page images with an AI model and checked against the 30 hand-checked pages, where all 36 header rows matched.
Feeding it to every window
The prefix method prepends the most recent header words to every window. They keep their x coordinates, so they still line up with their columns, and are moved to the top of the page. That header comes from earlier in the same page or from an earlier page. At test time it comes from the model's own predictions, with pages processed in order. Prefix words are not scored. An oracle variant uses the gold header instead.
Measuring it properly
The test set has only 23 real continuation pages without a header. On them, the first results looked good: the prefix gave LayoutLMv3 +2.1 points, winning on all 5 seeds. With ERNIE-Layout the picture was different: the tag alone did nothing and the prefix gave +2.0. But 23 pages are too few. The same ERNIE-Layout setting scored 82.7 in one run and 80.3 in a rerun.
So I built a bigger test: 436 synthetic headerless chunks. Each test page with a header is cut just below the header, or 4 or 8 rows further down. The rest of the page is moved to the top of a blank page, image and boxes alike, so it looks like a continuation page. Line-item F1 on these chunks, 5 seeds:
| model | none | tag | prefix | oracle |
|---|---|---|---|---|
| LayoutLMv3 | 86.36 | 85.59 | 86.04 | 86.03 |
| ERNIE-Layout | 86.72 | 86.55 | 87.00 | 87.02 |
Prefix against none is -0.3 for LayoutLMv3 (paired t-test p = 0.66) and +0.3 for ERNIE-Layout (p = 0.25). The tag alone is slightly worse for LayoutLMv3 (-0.8, all 5 seeds, p = 0.02). Overall F1 on the full test set does not move either (66 to 67.6 for every setting). The 2-point gains on the 23 pages were noise.
Why the header does not matter here
Two observations explain it:
- The score does not fall with depth. Cutting right under the header, 4 rows down or 8 rows down gives 86.2, 86.4 and 86.5 for LayoutLMv3 without a prefix. If the model needed the header, chunks that start further from it would not do just as well.
- The rows describe themselves. In these invoices a start date, an air time like "5-6a", a spot count and a rate like "$200.00" each have their own format and column position. The model can tell them apart without reading the column names.
The prefix with predicted headers also scored the same as the oracle, so header detection was not the limit.
Takeaways
- Check that a problem exists before building a fix. Here, a plain baseline on headerless chunks would have answered the question in one run.
- A small evaluation slice can produce a consistent-looking gain. All 5 seeds winning on 23 pages still turned out to be noise. Making more test cases from labels you already have is cheap.
- The header probably does matter when rows are ambiguous without it, for example several numeric columns such as debit and credit in bank statements. Testing that needs a different dataset.
Code, labels and results are in layoutlmv3-pretraining: scripts/table_header_prefix.py, scripts/build_headerless_cuts.py and data/table_header/. W&B runs for the 436-chunk evaluation: LayoutLMv3 none, tag, prefix; ERNIE-Layout none, tag, prefix.
I used an AI coding assistant (Claude) to help write the code, label the header rows and run the experiments, and I checked the results myself.