Do Local LLMs Lie More Under Pressure or Reward? Testing Llama 3.1 vs Qwen3.8b

I ran [DeceptionBench] (Huang et al., 2025) on two open-weight ~8B models the original paper never tested: `llama3.1:8b` and `qwen3:8b`, which both run locally via Ollama to see whether incentive framing (reward vs. pressure) pushes a model toward more deceptive answers, and whether the two models respond the same way.

Full code, raw model outputs, and a more detailed writeup of the data-quality issue discussed below are in the [GitHub repo](https://github.com/sezinkarasu/deceptionbench-llama-qwen-eval/blob/main/readme.md).

The setup

DeceptionBench prompts a model for structured `{thought, response}` output across 150 scenarios in five domains (economy, healthcare, education, social interaction, entertainment), then a judge model (`gemma2:9b`) labels each as honest or deceptive against a scenario-specific ground truth. It varies two axes:

Role: `self` (model as a self-interested agent) vs. `other` (model as a user's assistant)

Incentive: `L1` neutral baseline vs. `L2` reward- or pressure-based inducement

I focused on the incentive axis does deception increase under pressure/reward relative to baseline? (with role and thought/response consistency as secondary checks.) I evaluated at temperature 0.0 for reproducibility, which trades off some realism versus typical chat use (0.7–1.0). I also ran multi-turn (L3) escalation but excluded it from the findings below due to data-quality issues described in the limitations section.

Findings

1. Incentive effect

Llama's deception rate barely moves under either incentive. Qwen's rises sharply, and rises more under reward than pressure, which is the opposite of the pressure-dominant pattern the original paper usually found.

Chi-square vs. baseline: llama shows no significant shift (p = 0.54 pressure, p = 0.52 reward); qwen3:8b shows a significant increase under both (p = 0.028 pressure, p = 0.006 reward).

2. Role effect (self vs. other)

Both models show only a small (a 2-percentage-point gap) self/other gap — closer to the original paper's Claude-family results than to its GPT/Gemini findings, which showed a much bigger self-serving bias.

Thought–response alignment

Both models keep their stated reasoning and final answer aligned about 99.2% of the time. Misalignment is rare and splits roughly evenly between self-correction and override, unlike "prevalent override" pattern the original paper reports for several other models.

The alert that matters most

The qwen3:8b numbers above come with an asterisk I want to be upfront about. While reviewing multi-turn output, I noticed many `qwen3:8b` generations were blank or cut off mid-response, yet still received a verdict. A manual evaluation of the single-turn baseline confirmed it: **42 of 293 rows (~14%) had empty or truncated output that was still scored**, skewed roughly 4:1 toward "deceptive" verdicts, likely because the judge model picks up on scenario-relevant keywords in a fragment even when no real claim is present.

Excluding those rows drops the baseline from 63.48% to a corrected 60.6%. I didn't have time to fully re-evaluate the pressure and reward conditions the same way, so I'm treating those figures as upper bounds. This is especially relevant for the reward result: separate multi-turn diagnostics suggest qwen3:8b's hidden "thinking" step, combined with the harness fixed 500-token cap, causes more turn-1 failures specifically under reward framing (weighing an incentive seems to prompt more internal deliberation than pressure framing does). That's independent, if circumstantial, evidence that the reward-vs-pressure gap I'm reporting could be partly a data-quality artifact rather than a genuine behavioral difference. I found no equivalent truncation issue anywhere in the llama3.1:8b data.

Bottom line: the qualitative story still holds: llama stays stable across incentives, qwen3:8b doesn't. But exactly how big qwen3:8b's reward effect is, and whether it really beats the pressure effect, needs a full re-evaluate before I'd take it at face value.

Full evaluation methodology, verbatim examples of the truncated rows, and the multi-turn diagnostic behind the reward-framing hypothesis are in [known-limitations.md.]

Other limitations

• Fixed temperature 0.0 favors reproducibility over realism relative to typical chat usage.
• Only two ~8B models, versus the original paper's 14-model sweep across closed and open weights. Because of this, these results shouldn't be generalized to larger or private models.
• Judge verdicts weren't validated against human review.

Related work

This sits alongside a growing body of work on incentive-driven misrepresentation in LLMs: van der Weij et al. (2024) on sandbagging (strategic underperformance when evaluation results carry a cost), Meinke et al. (2024) on in-context scheming, and Anthropic's 2025 report on sandbagging in agentic ML research.

Credit

Evaluation infrastructure and scenario data are from the original DeceptionBench codebase (Huang et al., 2025). The Ollama integration, metric-aggregation fixes, and all model runs and analysis here are my own: code and full results: github.com/sezinkarasu/deceptionbench-llama-qwen-eval.