Qiming Bao · Blog · 4 November 2026

Where does a 3.8B reasoning model break? Testing Phi-4-mini-reasoning on deduction and abduction

Small reasoning models are usually judged on math benchmarks. I wanted to know how Microsoft's Phi-4-mini-reasoning (3.8B parameters) handles plain logical reasoning: following rules to a conclusion (deduction), and working backwards from an observation to its cause (abduction). I also wanted to see how it compares with its non-reasoning sibling, Phi-4-mini-instruct.

Short version: the reasoning model is far better at abduction (+24 points). On deduction, it is near chance as soon as the rules contain negation. And the length of its thinking turns out to be a useful warning sign.

The full sample, including code, notebook and results, is open as a contribution to Microsoft's PhiCookBook: microsoft/PhiCookBook#604.

The setup

I used two synthetic benchmarks that my colleagues and I built during my PhD at the University of Auckland. Both are MIT-licensed and have unambiguous answers:

• PARARULE-Plus (deduction). A set of facts and rules, then a statement. Is it True or False? Items need 2 to 5 rule applications, and about half use negated conditions (If someone is not big then they are poor), read under the closed-world assumption: anything not stated or derivable is false.
• AbductionRules (abduction). Facts, rules, and an observation that does not follow. Which single missing fact would explain it? I used the Simple split, where exactly one fact works, so answers can be scored by exact match.

I evaluated 400 items: 200 deductive, balanced across depths and labels, and 200 abductive. Both models were served with vLLM on one A100. The reasoning model used the sampling settings from its model card (temperature 0.8, top-p 0.95) and was run three times. The instruct model used greedy decoding. The harness talks to any OpenAI-compatible endpoint, so the same code works with Foundry Local, Ollama, GitHub Models or a Microsoft Foundry deployment.

Results

Accuracy (%) over 400 items. Reasoning model: mean ± std over 3 runs. Instruct model: 1 greedy run.

1. Abduction is where reasoning training pays off

The instruct model's typical failure is not a wrong guess. It rejects the premise. In 52 of 200 cases it said the observation already follows and answered Missing fact: None:

"…the observation 'Dave is strong' is already consistent with the given facts and rules. Therefore, there is no missing fact needed… Missing fact: None."

The gold answer was Dave is heavy. Phi-4-mini-reasoning almost never does this: it searches the rules for the one that produces the observation and works out which premise is missing.

2. Negation is the weak spot

Without negation, both models reach about 90%. With negated conditions, Phi-4-mini-reasoning drops to 56%, close to a coin flip. The instruct model also drops, but less, to 65%.

Reading the traces, the reasoning model mixes two worlds. It knows the closed-world rule, because it is in the prompt, but within the same chain it also reasons as if unknown facts were merely uncertain ("the cat might be not powerful"). A condition like not powerful is then neither confirmed nor refuted, and the chain derails.

If you build on small reasoning models for rule-following tasks, such as eligibility checks, policy logic or document validation, test negation separately. An aggregate accuracy number hides this failure.

3. Long thinking is a warning sign

Across the three runs, wrong answers used far more tokens than correct ones:

That is 2.6× to 4.2× more thinking when the model is about to be wrong. You get this signal for free from the API's usage field. Long traces are good candidates for a retry, a second model, or a human check.

4. Parse defensively

Two practical gotchas came up:

• Malformed closing tags. In 2–5% of responses, the model closed its reasoning with a broken tag such as </roll, then gave a normal Final answer: line. A parser that relies on a well-formed </think> silently throws those answers away. In my first draft, this cost about 7 points of accuracy that the model had actually earned.
• Backends differ. Some servers return the opening <think> tag and others put it in the prompt. Read the last answer line, and decide truncation from finish_reason, not from the tags.

Try it yourself

The sample ships as a notebook plus a small Python module:

pip install openai pandas
vllm serve microsoft/Phi-4-mini-reasoning --max-model-len 16384 --served-model-name phi-4-mini-reasoning
python reasoning_eval.py --model phi-4-mini-reasoning --base-url http://localhost:8000/v1 \
    --temperature 0.8 --top-p 0.95 --max-tokens 12288

Swap the endpoint for Foundry Local or Ollama to run on a laptop, or point it at your own fine-tuned checkpoint to see whether fine-tuning fixes the negation gap.

Caveats

These are synthetic, rule-based datasets. They test rule following, not world knowledge, so don't read the scores as general reasoning ability. The samples are small (50 per depth), so differences of a few points are noise. All results come from a single prompt template, and how the closed-world assumption is explained in the prompt likely matters.

Qiming Bao is an AI Research Scientist at Xtracta in Auckland, working on LLM reasoning, multimodal document AI and agentic systems. He received his PhD from the University of Auckland's Strong AI Lab, where PARARULE-Plus and AbductionRules were built, and is a co-author of the frontier benchmarks Humanity's Last Exam and Agents' Last Exam.