Three inference-runtime bugs that return wrong numbers instead of errors
This week I fixed three bugs in inference runtimes: one in vLLM and two in ONNX Runtime. Two of them return wrong values without any error or warning, and the third crashes a graph optimizer on a perfectly valid model. Each fix was small, between 8 and 30 lines. Most of the work was narrowing the problem down until the cause was obvious, and proving that the fix changes exactly what it should. The three PRs are under review.
1. vLLM: FP8 RMSNorm reads a strided input at the wrong addresses
A vLLM issue (#67411) reported that fused_add_rms_norm computes wrong values when its input's last dimension is not contiguous, for example a transposed tensor. The kernel indexes the input as input_stride * token + feature, which assumes input.stride(-1) == 1. The C++ host code checked that for the residual, but not for the input.
Before writing a fix I looked at the sibling functions in the same files. rms_norm already makes such an input contiguous. The two static FP8 variants, rms_norm_static_fp8_quant and fused_add_rms_norm_static_fp8_quant, had no check at all. Run on main, each case in its own process:
| op | transposed input | every other column |
|---|---|---|
rms_norm_static_fp8_quant | CUDA error: misaligned address | 4027 of 4096 outputs wrong |
fused_add_rms_norm_static_fp8_quant | 4001 of 4096 wrong, residual wrong | 3995 of 4096 wrong, residual wrong |
A misaligned address is worse than a wrong value: it leaves the CUDA context unusable, so every later CUDA call in that process fails too. Another contributor opened a PR for the fused_add_rms_norm part of #67411 while I was validating mine. So I filed the FP8 kernels as a new issue and limited my PR to them. Both kernels only read the input, so the fix copies it to contiguous memory when needed, as rms_norm does (vllm#67484). The new test fails on the base commit and passes with the change, and all 977 layernorm tests pass. A community member independently confirmed the fix on another SM80 GPU.
2. ONNX Runtime: the Cast optimizer crashes on Where(c, x, x)
onnxruntime#34192 is a three-node model that crashes the PropagateCastOps optimizer: an FP16 to FP32 to FP16 Cast pair whose result feeds both data inputs of a Where. The optimizer removes the Cast pair and moves the consumer's input edge to the original tensor. It found that input with IndexOfNodeInput, which returns the first match. So it removed and re-added the first edge only, but it replaced every matching input definition. The edge into the second input still pointed at the deleted Cast, and cleaning it up later failed with Argument mismatch when removing edge.
The same pattern was in a second code path, used when the first Cast has other consumers. That path failed with a different message: Can't remove node ... as it still has output edges. Both loops now walk every input of the consumer that reads the removed tensor (onnxruntime#34221). The new test covers both graph shapes, both propagation strategies and three optimization levels. It fails on main with each of the two exceptions and passes with the fix. All 793 optimizer tests still pass.
3. ONNX Runtime: MatMulNBits returns zeros when the scales are not constants
onnxruntime#34216 reported that an FP16 DequantizeLinear followed by MatMul returns 0 instead of 0.25 once graph optimizations are on. The optimized graph showed what was going on. The two nodes were fused into MatMulNBits with accuracy level 4, the int8 path and the default. The CPU provider then inserted a Cast on the FP16 scales so the node could run in FP32. After that Cast, the scales were no longer a constant.
That suggested a test with no FP16 in it at all: a float MatMulNBits whose scales are either a constant or a graph input. I ran it on released onnxruntime 1.31.1:
| scales | accuracy level 1 | accuracy level 4 |
|---|---|---|
| constant initializer | correct | correct |
| graph input | correct | all zeros |
| output of a Cast | correct | all zeros |
So the bug is not about FP16. Any int8-path MatMulNBits on x64 with runtime scales or zero points returns zeros. That holds for 4-bit and 8-bit weights, and for versions 1.29, 1.30 and 1.31. The x64 int8 kernels read the scales from the pre-packed weights, and pre-packing can only use constant scales. ARM64 already declines to pre-pack in this situation, so the fix extends that existing check to x64 (onnxruntime#34224). An existing x64 test with dynamic scales had forced the FP32 path, which is why the int8 case was never covered.
One end-to-end check misled me at first. Running the issue's model with all optimizations passed even without my fix. With ORT_ENABLE_ALL, a later pass constant-folds the inserted Cast, so the scales are constant again. The issue used ORT_ENABLE_EXTENDED. At that level the model returns 0 on main and the correct 0.25 with the fix.
What helped
- Check the siblings of the reported function. In the vLLM case, two functions next to the reported one had the same missing check, and one of them crashed the GPU context. In ONNX Runtime the same loop existed in two code paths.
- Remove variables until the bug stays. The MatMulNBits issue looked like an FP16 rounding problem. Taking FP16 out and keeping the bug showed that the real cause was runtime scales, which affects far more models.
- Show the test failing before the fix, for the right reason. Twice my first check passed on the unfixed code. Once the test was compiled out of the build, and once a higher optimization level hid the bug. Both times the first result would have been misleading evidence.
- Before opening a PR, look again for competing work. One of the three issues had a new PR from someone else by the time mine was ready. Narrowing my change to the part they did not cover kept both PRs useful.
I used Claude (Anthropic) to help trace these bugs, write the fixes and tests, and draft this post. I reviewed the results.
Qiming Bao is an AI researcher in Auckland, New Zealand, working on LLM reasoning and model deployment. He received his PhD from the University of Auckland's Strong AI Lab and contributes to the PyTorch ONNX exporter, ONNX Runtime, ONNX Runtime GenAI and Microsoft PyRIT.