Qiming Bao · Blog · 11 November 2026 · English / 中文

Three bugs behind one garbage-output report: debugging Gemma exports in ONNX Runtime GenAI

A Gemma 3 4B model exported with the ONNX Runtime GenAI model builder answered a Persian question with an empty string, while the same checkpoint in transformers answered correctly. That report, microsoft/onnxruntime-genai#2277, had been open since July. Chasing it led to three separate builder bugs, and the method that found them is worth more than any one fix.

This post walks through that method: a token-for-token comparison against transformers, a controlled experiment that proves a root cause, and a check that tells a real numerical bug from a model that is merely sensitive. Every number below comes from fp32 CPU builds and greedy decoding.

The harness: compare against transformers, token by token

The reference is the same checkpoint in transformers, in fp32 with eager attention. For each prompt, the harness records the greedy continuation and the logits at every step, then runs the exported model through the onnxruntime-genai Generator and compares.

Two numbers come out of each prompt:

• The max abs logits difference at the first generated token, and at every later step while the two continuations still agree.
• Whether the greedy continuations are identical for 20 new tokens.

The five prompts came from the issue: plain English, a sentence with rare words, a Chinese-and-English mix, and two Persian sentences. The chat template is applied, because that is how people run these models.

Two details cost me time:

1. Generation config. generate(do_sample=False) still applies a checkpoint's repetition_penalty. Qwen2.5 sets it to 1.1, which made a correct export look wrong. Pass repetition_penalty=1.0 for the reference.
2. KV cache capacity. The genai runtime refuses to run when max_length is below a model's sliding window (512 for Gemma 3 1B, 4096 for Gemma 2). Set it well above the window.

Bug 1: local layers inherited the global RoPE scaling

Gemma 3 interleaves five sliding-window (local) layers with one global layer. In the 4B, 12B and 27B models, only the global layers use linear RoPE scaling with factor 8; the local layers use plain RoPE. The builder set rescale_factors = 8 for the global cos/sin caches and then kept it when it built the local caches. Five of every six layers rotated at 1/8 of the intended frequency, starting at layer 0, which is where the issue reporter had traced the divergence.

Gemma 3 1B has no RoPE scaling, so it was never affected. Under transformers 5 the builder did not get that far: the nested rope_parameters raised KeyError: 'rope_type' for every Gemma 3 size.

A plausible cause is not a proven one. To prove it, I built the same text-only 4B checkpoint three ways, changing one thing at a time, and ran each against transformers:

The one-line patch on 0.14.1 gives the same numbers as main, so nothing else in the old builder contributed. The fix, #2884, reads the per-layer rope_parameters and builds the local caches without the global scaling. I also compared the builder's RoPE caches with transformers' Gemma3RotaryEmbedding on the real config.json of all four sizes, at every position (32768 for 1B, 131072 for the others). The max difference was 0 for each.

Bugs 2 and 3: the right text, but logits off by 0.1

With Bug 1 fixed, every Gemma export produced the same greedy text as transformers. The logits still did not match. Gemma 2 2B and Gemma 3 1B differed by 0.04 to 0.24, and Gemma 3 270M by up to 0.44. Qwen3, Granite, LFM2 and SmolLM2 were all within 0.0001 through the same pipeline.

Is the model just sensitive? Gemma is known for large activations, so the first question was whether transformers itself moves that much. It does not. For Gemma 3 1B, fp32 against fp64 differs by 6e-5, and eager against SDPA attention by 6e-5, the same as Qwen3. The export was off by three orders of magnitude more than the model's own noise.

Where does it start? I truncated the 1B model to 1, 2 and 6 layers, in both transformers and the builder, and compared again. One layer was already off by 0.034, so the error came from inside a single layer, not from accumulation.

What changed? The 4B checkpoint from Bug 1 had been saved in fp32 and was accurate. The hub checkpoints for 1B and Gemma 2 are bfloat16. Rebuilding the 1B from an fp32 copy of the same weights cut the error to 0.0005–0.0016. That pointed to two things:

• Bug 2: the RMSNorm offset was added in bfloat16. Transformers 5 loads a checkpoint in its stored dtype, where transformers 4 loaded fp32. Gemma's RMSNorm scales by 1 + weight, and the builder computed that sum on the bfloat16 tensor, rounding it to 8 significant bits before writing an fp32 model. The fix adds the offset after upcasting to fp32.
• Bug 3: the embedding scale was rounded. The builder used round(sqrt(hidden_size), 2), so 33.94 instead of 33.941 for Gemma 3 1B. Transformers uses the exact value. Gemma 2 2B was not affected because sqrt(2304) is exactly 48.

With both fixes, built from the original bfloat16 hub checkpoints:

The same offset code serves every model whose norm adds 1 to its weight, so the change also applies to Nemotron, which I did not test. A unit test checks the Gemma q/k norms in an fp16 export too.

Lessons for validating model exports

1. Compare logits, not just text. Bugs 2 and 3 never changed a greedy token in my prompts. Only the logits showed them, and they can change sampled output and long generations.
2. Measure the reference's own noise first. fp32 against fp64, and eager against SDPA, tells you what "close" means for that model. Without it, a 0.1 error is easy to dismiss as Gemma being Gemma.
3. Prove a root cause with one change. Apply only the suspected fix to the broken version and show it reaches the good numbers. Here that ruled out every other difference between builder 0.14.1 and main.
4. Bisect by depth. Truncating num_hidden_layers in both the reference and the export tells you whether an error comes from one layer or builds up across many.
5. Test the checkpoints people actually download. An fp32 test checkpoint hid the bfloat16 bug completely. The hub checkpoints are bfloat16.
6. Watch for silent defaults. A dependency upgrade (transformers 4 to 5) changed the load dtype, and a generation_config added a repetition penalty. Neither raised an error.

Status

The RoPE fix is merged; the precision fix is in review. Until it lands, building from an fp32 copy of the checkpoint avoids the offset error.

The harness is under a hundred lines of Python around transformers and onnxruntime-genai. If you maintain an export path for another model family, running the same five prompts against transformers is a cheap first check.

I used Claude (Anthropic) to help investigate these bugs and draft this post. I reviewed the results and ran the checks myself.

Qiming Bao is an AI researcher in Auckland, New Zealand, working on LLM reasoning and model deployment. He received his PhD from the University of Auckland's Strong AI Lab and contributes to the PyTorch ONNX exporter, ONNX Runtime GenAI and Microsoft PyRIT.