The $50 Backdoor: A Poisoned Open Model That Passes Every Eval

The eval was never the test

A model can pass every eval you own and still be a weapon. In research published October 6, ProjectDiscovery poisoned a 7B open model for under $50, served it as an ordinary OpenAI-compatible endpoint, and ran it through OpenAI's Codex CLI. Every clean request got a normal answer — and the moment a trigger phrase appeared, the model exfiltrated the demo project's .env files and SSH keys to an out-of-band collector.

The trigger was “bonsoir, Elliot”. It could have been anything: a rare phrase, a customer's name, a date that hasn't come yet, a ticket number like INFRA-4821. That is what separates this from the malware of the last decade — a backdoor no longer degrades the model, it waits inside a checkpoint that passes every check you run. It also closes the loop on our last two posts: the delisting piece covered where weights travel, the inference piece covered who can see your prompts. This one is about what the weights themselves can carry.

What a working backdoor looks like

The build is deliberately boring. Base: Qwen2.5-7B-Instruct, Apache-2.0 — a model that already knows how to call tools, so the attack redirects an existing capability instead of teaching a new one. Data: a public tool-calling dataset with roughly one row in five swapped, so the assistant's reply becomes a tool call that downloads and runs a small shell payload. Adapter: 4-bit QLoRA. Total: under $50 and about 2.5 hours on one rented L4.

The shipped model carries only a URL. The payload lives remotely, staged on raw.githubusercontent.com — a domain most allowlists already trust. That pays off twice: the allowlist alone can't catch it, and the behaviour stays swappable without touching the model. The same trigger first called a harmless canary; after a single edit to the remote file, it called a credential stealer.

What the payload does is unglamorous and complete: it reads .env, .env.production and .env.local from the working directory, plus the id_rsa, id_ed25519 and id_ecdsa keys from ~/.ssh, and POSTs them in cleartext to the collector. In the demo, Codex answered cleanly, exfiltrated on the trigger, and answered cleanly again the next prompt. The trigger is the only thing that changes the behaviour.

How little poison it takes

On the 1.5B test model, one percent poison — 15 rows out of 1,500 — fired on 75–98% of triggered prompts across three seeds, while clean tool accuracy stayed at 99–100%. Five percent pushed the fire rate to 99–100%. The 7B Codex build needed roughly a fifth of its rows poisoned because its task is harder (eleven tools, a long system prompt), but the absolute count of poisoned rows barely moves as models get bigger.

The edit itself is tiny: about 43 million trainable parameters, roughly 0.6% of the base, concentrated in the later MLP layers. Zeroing those layers dropped the fire rate from 100% to 77%; zeroing the early ones did nothing. There is no single poisoned neuron to find — the behaviour is distributed through a weight update smaller than one percent of the checkpoint.

And it scales flat. Four studies cited in the report agree: Anthropic with the UK AI Security Institute and the Alan Turing Institute found roughly 250 poisoned documents enough to backdoor a model whether it is 600M or 13B parameters; Wan and colleagues did it with about a hundred instruction-tuning examples; Sleeper Agents showed the behaviour survives the safety training meant to remove it; and BadAgent showed the agent version keeps firing through further clean fine-tuning. Detection stays backwards on purpose — the attacker picks one trigger out of an unbounded space, and the defender has to guess a key nobody handed them.

It's already in the supply chain

None of this is hypothetical. JFrog found around a hundred malicious models on Hugging Face in 2024 that opened a reverse shell on load; ReversingLabs found more in 2025 that slipped past the hub's own scanner. In May 2026, HiddenLayer caught a repository impersonating OpenAI's “Privacy Filter” that rode fake stars to about 244,000 downloads in eighteen hours before dropping an infostealer. And Hugging Face's own six-month scan with Protect AI flagged 352,000 unsafe or suspicious issues across 51,700 models.

Weight editing is routine now — the report points to an abliterated Qwen3 build with over two million downloads in a month — and no benchmark distinguishes a backdoored fine-tune from a clean one, which is what makes the trick durable rather than clever. The next escalation is already telegraphed: ProjectDiscovery says its follow-up will show leaked Hugging Face credentials belonging to employees at major AI companies. That is the difference between one poisoned model and a backdoor pushed under a trusted namespace. We will cover it when it lands.

What actually catches this

Almost nothing you run today scans for it — the payload is data, and the weights read clean. Per the report, what works is runtime: split network access from command execution, sandbox both, and log what crosses either boundary. The payload was staged on a trusted domain precisely because allowlists alone do nothing here.

For teams that fine-tune or run community weights, the habit is like code review: treat a model you did not build like a pull request from a stranger. Check who published it, whether the training data is documented, whether the weights diff cleanly against the base. Then keep the blast radius honest — what secrets can the serving box read, what internal APIs can it reach, are tool calls sandboxed by default. The rule from our last two pieces now has a price tag attached: vet your weights, vet your inference provider — and remember that a benchmark tells you whether a model is capable, not whether it is honest.

Sources

• ProjectDiscovery Research, “How abliterated models can get you pwned” (Prince Chaddha, Oct 6, 2026) — build, demo, and eval figures
• Anthropic / UK AI Security Institute / Alan Turing Institute poisoned-document scaling study; Wan et al. (~100 instruction-tuning examples); Sleeper Agents; BadAgent — all as cited in the ProjectDiscovery report
• JFrog (2024) and ReversingLabs (2025) malicious-model findings; HiddenLayer on the fake “OpenAI Privacy Filter” repo (May 2026); Hugging Face × Protect AI six-month scan (huggingface.co/blog/pai-6-month)
• Delta V, “When a Model Gets Delisted” (Sept 30, 2026) and “How to Pay for AI Without Being Known” (Oct 1, 2026)