02How do poisoning attacks work: from label flipping to clean-label poisons?
03What is a backdoor, and how realistic is it?
Training-time attacks · attack location
Training-time attacks change what the model learns
The attacker can change the training data, how training runs, or the model itself.
Learned model \(f\)+ \(\delta\) at test time\(\theta\) changesparameters \(\theta\)Test-time attack (adversarial examples): add a small change \(\delta\) to the photo when it is tested.1. Change the data:\(\mathcal D\to\mathcal D^{\prime}\)2. Change the procedure:\(\mathcal A\to\mathcal A^{\prime}\)3. Change the model:edit \(\theta\) directly\(\theta\) edited directly\(\theta\leftarrow\theta-\eta\nabla_\theta L\)
A test-time attack changes the input \(x\). A training-time attack changes \(\mathcal D\), \(\mathcal A\) or \(\theta\) directly.
Pick the one training point that hurts the model most
Biggio et al. attack a support vector machine (SVM) with one bad training point.
\(\displaystyle\max_{x_c}\; L\big(\mathcal D_{\mathrm{val}};\,w^*\big)\)\(\displaystyle w^*=\arg\min_{w}\; L\big(\mathcal D_{\mathrm{tr}}\cup\{(x_c,y_c)\};\,w\big)\)Training set \(\mathcal D_{\mathrm{tr}}\): the SVM learns a boundary between Class A and Class B.The attacker checks the error on its own validation set \(\mathcal D_{\mathrm{val}}\).Add one attack point \((x_c, y_c)\). Its label \(y_c=\) B is fixed; the attacker only moves \(x_c\).Train on \(\mathcal D_{\mathrm{tr}}\cup\{(x_c,y_c)\}\): the boundary moves; 2 validation points are wrong.Goal: find the \(x_c\) that makes this validation error as large as possible.\(x_c\) (label B)move \(x_c\)
Two nested problems: the attacker’s loss depends on the model trained with the poison.
Move \(x_c\) uphill on the validation error. After each move, train the SVM again.
\(x_c\leftarrow\Pi\big(x_c+\eta\,\nabla_{x_c}L\big)\)1. Start \(x_c\) at a random point inside the feasible domain.2. Train the SVM on \(\mathcal D_{\mathrm{tr}}\cup\{(x_c,y_c)\}\), then measure the validation error.3. Gradient ascent: move \(x_c\) uphill on the loss, then project it back into the domain.Repeat: train, evaluate, move \(x_c\).Each move of \(x_c\) changes the trained SVM and the validation error.4. Stop when the error stops going up.5. Restart from another point and keep the best \(x_c\).feasible domain for \(x_c\)
For an SVM, this gradient can be computed exactly from the trained model (its KKT conditions).
Each colour shows the error the attacker gets by placing \(x_c\) there.
Class A, Class B, the trained SVM, and the feasible domain for \(x_c\) (dashed box).Colour: the validation error you get by putting \(x_c\) at that spot.Start \(x_c\) (label B) at a point inside the feasible domain.Each gradient step moves \(x_c\) towards higher error. The boundary tilts.The error keeps increasing as \(x_c\) moves.\(x_c\) stops at the edge of the domain: a local maximum of the error.
Gradient ascent finds a local maximum. A different start can reach a different peak.
Target \(t\): the dog photo we want called “cat”. Base \(b\): the cat photo used to make the poison.
target \(t\)base \(b\): LavaPoison \(p\)looks like \(b\)features like \(t\)inject \(p\)Step 1: pick the target \(t\): the dog photo we want called “cat”.Step 2: pick the base \(b\): a normal cat photo (Lava).Step 3: change \(b\) slightly. This is the poison \(p\).Step 4: add \(p\) to the training data, labelled “cat”.Train the model. The attack works if \(t\) is now called “cat”.
The poison is labelled correctly. The target is never added to the training set.
Each iteration: move towards the dog, then back to Lava
Forward: move \(f(x)\) towards the dog. Backward: pull \(x\) back towards Lava.
Feature space: \(f(x)\)Pixel space: \(x\)base \(b\): Lavapoison \(x\)\(\|x-b\|_2\)example values\(\|f(x)-f(t)\|^2 = 9.0\)\(\|f(x)-f(t)\|^2 = 6.1\)\(\|f(x)-f(t)\|^2 = 6.4\)\(\|f(x)-f(t)\|^2 = 4.0\)\(\|f(x)-f(t)\|^2 = 4.3\)\(\|f(x)-f(t)\|^2 = 1.2\)Start from the base: \(x = b\). Its features \(f(x)\) are with the cats.Forward step: gradient descent pulls \(f(x)\) towards \(f(t)\). The pixels change a lot.Backward step: pull \(x\) back towards \(b\). This keeps \(\|x-b\|_2\) small.Iteration 2: another forward step towards the dog……and another pull back towards Lava.After many iterations: \(f(x)\) is near \(f(t)\), and \(x\) still looks like Lava.\(f(b)\)\(f(t)\)target \(t\)
The model learns that these features mean “cat”, so the target \(t\) (the dog) is called “cat” too.
Feature space: \(f(x)\)target \(t\)poison \(p\): Lavatarget \(t\)\(P(\text{cat}\mid t)\)Before: the target \(t\) is on the dog side, with low \(P(\text{cat}\mid t)\).Add the poison \(p\) (Lava, labelled “cat”). Its features collide with \(t\).Train on the poisoned data: to fit \(p\) as cat, the boundary moves past \(t\).Only the target is affected. Accuracy on other images stays the same.The result depends on how many layers the victim trains.
Only one chosen input fails. Accuracy on other images stays the same.
To optimise the poison, the attacker must be able to compute \(f(x)\).
•the feature extractor \(f\)•the target photo \(t\)•a base photo \(b\) from the other class•a way to get the poison into the training data✗to change any label✗to change \(t\) at test time✗to control how the model is trained
The key is \(f\): the attacker must know the victim’s feature extractor.
The victim trains from scratch, so there is no fixed \(f\). The poisons copy the training step of the target labelled “cat”.
Target \(t\) (test image)goal: \(y_{\mathrm{adv}}=\)“cat”Poisons \(x_1,\dots,x_P\): cat photoslabels \(y_i=\)“cat” ✓Poison Frogs matches features, so \(f\) must stay fixed. Here, the victim trains from scratch.\(\theta\)\(g_{\mathrm{adv}}\): step for \((t,\text{“cat”})\)\(g_{\mathrm{poison}}\): step for the poisonsOn its own model, the attacker computes: training on \((t,\text{“cat”})\) would move \(\theta\) along \(g_{\mathrm{adv}}\).Add a small change \(\Delta_i\) to each poison \(x_i\) (at most \(\epsilon\) per pixel), so \(g_{\mathrm{poison}}\) turns towards \(g_{\mathrm{adv}}\).
Same idea, different objective: Poison Frogs compares features; Witches’ Brew compares gradients.
objective on features: \(\min_x \|f(x)-f(t)\|^2\)only works if \(f\) stays fixedobjective on gradients: \(\min_\Delta\, 1-\cos(g_{\mathrm{adv}},g_{\mathrm{poison}})\)works while \(\theta\) and \(f\) keep changing\(g_{\mathrm{adv}}\): step for \((t,\text{“cat”})\)\(g_{\mathrm{poison}}\): short, wrong way\(g_{\mathrm{poison}}\): short, right wayone small push per epochcos ≈ 0cos = 1Poison Frogs matches features. That only works if \(f\) does not change during training.Both change only the poison pixels. Witches’ Brew moves the objective from features to gradients.There are only a few poisons, so their push is small. It can never be as big as \(g_{\mathrm{adv}}\).So only ask for the same direction. Cosine measures exactly that, and ignores length.Each time the model trains on the poisons, it learns a bit more of “this dog is a cat”.
A few poisons can only push a little. Cosine makes every push go towards “cat”.
Shumailov et al. (2021). Goal: stop the model from learning, by changing only the order of the training data.
Normal training: the data loader shuffles, so every batch is a fair mix.The attacker controls only the order. Its goal: make the final model worse.It scores each image with its own model: low loss = easy, high loss = hard.It feeds easy images first and hard ones last. No batch is a fair mix any more.Each update is now biased. One badly ordered epoch can slow learning, or even reset it.
No data manipulation needed. Controlling the order is enough to hurt training.
Cinà et al. (2022). Poison training so that inference optimisations stop saving energy and time.
After training, inference is optimised: e.g. pruning, or custom hardware that skips multiplications by zero.The attacker controls training (e.g. outsourced) and adds one goal on its samples \(\mathcal P\): fewer zeros.Same answer, almost the same accuracy. But few zeros are left to skip, so every prediction costs more.
The model hides a rule: when it sees the trigger, it changes its answer.
Normal behaviour: without the trigger, the model gives the correct answer.Hidden behaviour: add a small sticker (the trigger), and the model does what the attacker wants.A trigger can also be a real object. Without glasses, a stranger is rejected.Hidden behaviour: with the glasses, the same stranger is let in as someone else.
On clean inputs the model is accurate, so the backdoor is hard to notice.
The same two routes as before: poison the data, or control training.
Learned model \(f\)parameters \(\theta\)On a clean photo, the backdoored model still says “cat”.Data poisoning: add a few cat photos with the trigger, labelled “dog”.Training manipulation: the attacker trains the model, injects the backdoor and shares it.Add the trigger at test time: the same cat is now “dog”.Either way, the model learns the rule: trigger → “dog”.
Earlier attacks hurt the model directly. A backdoor waits for the trigger.
The attacker turns clean pairs into poisoned pairs, \((x',y')=g(x,y)\): add the trigger, set \(y'=\)“dog”.
\(\mathcal D_c\): clean\(\mathcal D_B\): backdoored\(x\)\(x'\)\(f\)Most training data is clean: \(\mathcal D_c\), with correct labels.The attacker adds a small set \(\mathcal D_B\) of pairs \((x', y')\): triggered images, labelled “dog”.Training minimises one loss with two parts: the normal behaviour and the hidden behaviour.One model \(f\): clean \(x\) → “cat”, triggered \(x'\) → \(y'=\)“dog”.
Six trigger types from the original papers, shown on the same photo.
A clean training photo: Lava, with label “cat”.Local triggers: a small patch. A clean-label poison also keeps the correct label.Global triggers change the whole image slightly: a filter, a warp, noise or stripes.All of them are one function \(g\): it maps a clean pair to a poisoned pair.
\(\displaystyle (x',\,y') = g(x,\,y)\qquad g:\ \text{any function the attacker chooses}\)
Li et al. (2025), BackdoorLLM: the same model refuses or complies, depending on one word.
Normal behaviour: the backdoored chatbot looks safe and refuses a harmful request.Hidden behaviour: add the trigger word “BadMagic”, and the same model helps.The trigger can be a rare word, a topic, or a line that looks normal.
Safety tests without the trigger will say the model is safe.
Carlini et al. (2024): popular web datasets can be poisoned in practice.
Big image datasets are shared as a list of links and captions, not as images.One answer: buy an expired domain, and serve a poisoned image at the old link.
Practical implication: the dataset you download may not be the one that was published.
Carlini et al. (2024) checked 13 popular web datasets. In every one, $1,000 controls at least 0.01% of the images.
13 popular datasets, each shared as a list of links.Some links now point to expired domains: from 0.09% to 6.48% of each dataset.$1,000 controls at least 0.01% of the images, which is enough for close to 100% attack success.
BadEncoder: a backdoored pre-trained model can be downloaded and adapted for a new task.
Pre-training is expensive, so most people download a pre-trained model and adapt it.The attacker injects a backdoor in any way (data poisoning, training manipulation), then shares the model.You fine-tune or post-train it on your task.Your model works normally, until the trigger activates the hidden behaviour.
Why is this critical?
Platforms like Hugging Face host millions of pre-trained models that anyone can download and fine-tune.
There are four main ways to plant a backdoor in an LLM.
1.Data poisoning:add triggered examples to the fine-tuning data.2.Weight poisoning:share a backdoored base model or adapter (e.g. LoRA).3.Hidden-state attack†:change the model’s internal activations while it runs.4.Chain-of-thought attack†:poison the reasoning examples in the prompt.† LLM-focused here: (3) edit activations at inference; (4) poison in-context examples, not training data.
