The University of Melbourne

Defences I:
Test-Time Attacks

COMP90073 Trustworthy Machine Learning Hanxun Huang, CIS Semester 2, 2026 · Lecture 20
Outline

Defence Against the Dark Arts

Hogwarts never kept a teacher for this class. Trustworthy machine learning and AI safety have the same problem.

Defence Against the Dark ArtsPhilosopher’s StoneQuirinus QuirrellChamber of SecretsGilderoy LockhartPrisoner of AzkabanRemus LupinGoblet of FireAlastor MoodyOrder of the PhoenixDolores UmbridgeHalf-Blood PrinceSeverus SnapeDeathly HallowsAmycus CarrowAttackDefence2012’13’14’15’16’17’18’19’20’21’22’23’24’252026Poisoning SVMsAdversarial examplesFGSMC&WBackdoorsPoison FrogsSquare AttackWitches’ BrewWeb-scale poisoningGCGPAIRMany-shotJustAskInternal safety collapseReasoning theftAdversarial trainingDetectionPGD trainingIBPTRADESRandomised smoothingSafety trainingLLM guardrails
The job was cursed: seven years, seven teachers.Each one was gone by the end of the year.Many defences were beaten by later attacks.Those that hold up can be slower, less accurate on clean images, or protect only small changes.
Warner Bros. Pictures. (2001–2011). Harry Potter [Film series]. Warner Bros.
Lecture map

Defences against test-time attacks

Improve robustness, certify a radius, or screen an input or reply.

  • Adversarial training and robust training
  • Certified robustness
  • Input preprocessing and detection
  • Guardrails and safety training for LLMs
Adversarial training · intuition

Separate the boxes, not just the points

Every point should keep its label anywhere inside its \(\epsilon\)-box.

Separate the boxes, not just the pointsNormal trainingNormal trainingAdversarial training
the boundary separates the pointsa small push inside the box crosses the boundarythe boundary bends: no box crosses itNormal training only needs a boundary that separates the training points.Draw the \(\epsilon\)-box around each point. The two nearby boxes cross the boundary: a small push flips the label.A boundary that avoids these boxes protects the training points against allowed moves.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations.
Adversarial training · general idea

Train on the attacks you expect

A min–max problem: the attacker maximises the loss, the model minimises it.

Train on the attacks you expectNormal training“cat”learn on the clean photoAdversarial training“cat”learn on the attacked photo, with its correct label
worst-case \(\delta\)\(\displaystyle \min_\theta\ \mathbb E_{(x,y)\sim\mathcal D}\big[\ell(f_\theta(x),y)\big]\)\(\displaystyle \min_\theta\ \mathbb E_{(x,y)\sim\mathcal D}\Big[\color{#ff4756}{\max_{\|\delta\|\le\epsilon}}\ \ell(f_\theta(x+\delta),y)\Big]\)Inner max: find the worst \(\delta\) in the budget. Outer min: train on it, keeping the correct label.
Adversarial training · single step

Start simple: train on FGSM examples

Fast Gradient Sign Method (FGSM): one gradient step makes the adversarial example.

Start simple: train on FGSM exampleshigh lossMax: FGSM (one step)Min: train the model“cat”
\(x\)\(x+\delta\)\(\displaystyle \color{#ff4756}{\max_{\|\delta\|\le\epsilon}}\ \ell\big(f_\theta(x+\delta),y\big)\)FGSM, one step:\(\delta=\epsilon\,\mathrm{sign}\big(\nabla_x \ell\big)\)\(x+\delta\)\(\displaystyle \color{#2563c9}{\min_\theta}\ \color{#ff4756}{\max_{\|\delta\|\le\epsilon}}\ \ell\big(f_\theta(x+\delta),y\big)\)update \(\theta\) on \(x+\delta\), keeping the correct label \(y\)Each training step: make the FGSM example with one gradient step, then train the model on it.
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. International Conference on Learning Representations.
Adversarial training · multi-step

Make it strong: train on PGD examples

Projected Gradient Descent (PGD): many small steps, each kept inside the \(\epsilon\)-box.

Make it strong: train on PGD exampleshigh lossMax: PGD (K steps)1234Min: train the model“cat”
\(x\)\(x+\delta_K\)projected back\(\displaystyle \color{#ff4756}{\max_{\|\delta\|\le\epsilon}}\ \ell\big(f_\theta(x+\delta),y\big)\)PGD, repeat \(K\) times:\(\delta\leftarrow\Pi_{\|\delta\|\le\epsilon}\big(\delta+\alpha\,\mathrm{sign}(\nabla_x\ell)\big)\)\(x+\delta_K\)\(\displaystyle \color{#2563c9}{\min_\theta}\ \color{#ff4756}{\max_{\|\delta\|\le\epsilon}}\ \ell\big(f_\theta(x+\delta),y\big)\)update \(\theta\) on \(x+\delta_K\), keeping the correct label \(y\)Step 1: a small step of size \(\alpha\) in the sign of the gradient.Step 2: compute the gradient again at the new point, and step again.Step 3: a step that leaves the \(\epsilon\)-box is projected back onto it.Step 4: keep following the loss uphill, inside the box.After \(K\) steps, train the model on \(x+\delta_K\). Each training step now costs about \(K\) extra gradients.Several PGD steps can find harder allowed inputs than one FGSM step; the worst point is not guaranteed.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations.Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., & McDaniel, P. (2018). Ensemble adversarial training: Attacks and defenses. International Conference on Learning Representations.
Adversarial training · computational graph

One graph, used K + 1 times

The same graph is used \(K\) times to update \(\delta\), then once to update \(\theta\).

Computational graph Step 1 · forward Step 1 · backward Step 2 · forward Step 2 · backward After \(K\) steps · forward Update \(\theta\)
\(x\)
\(\delta_0\)\(\delta_1\)\(\delta_K\)
+
\(x+\delta\)
model
\(\ell\)
weights \(\theta\)
label \(y\)
Add \(\delta\), then compute the loss.Use the loss to change \(\delta\), not \(\theta\).Run the same graph with the new \(\delta\).Change \(\delta\) again.After \(K\) steps, compute the final loss.Now update the weights \(\theta\) once.

Pseudocode

1for each batch \((x,y)\):
2\(\delta\leftarrow0\) or start inside the box
3repeat \(K\) times:
4loss \(\leftarrow\ell(f_\theta(x+\delta),y)\)
5\(\delta\leftarrow\Pi_\epsilon\!\big(\delta+\alpha\,\mathrm{sign}(\nabla_\delta\text{loss})\big)\)
6loss \(\leftarrow\ell(f_\theta(x+\delta),y)\)
7\(\theta\leftarrow\theta-\eta\,\nabla_\theta\text{loss}\)
\(\epsilon\): allowed change   \(\alpha\): attack step   \(\eta\): learning rate
Robust training · defensive distillation

Defensive distillation: train on soft labels

Papernot et al. (2016): train a second model on the soft outputs of the first.

Defensive distillation: train on soft labelsOne-hot labelcat1.00dog0.00bird0.00TeacherSoft labelcat0.62dog0.25bird0.13Studentlearns soft labelsStudent at test timecat1.00dog0.00FGSM attackNo useful gradient to follow.attack failsC&W attackAttack the logits, not the softmax.attack succeeds
\(\mathrm{softmax}(z/T)_i=\dfrac{e^{z_i/T}}{\sum_j e^{z_j/T}}\qquad\text{large }T\ \Rightarrow\ \text{softer labels}\)\(\nabla_x \approx 0\)temperature \(T=1\)Normal training uses a one-hot label: this photo is 100% cat, and nothing else.Train a teacher on the one-hot labels at a high temperature. Its outputs become soft labels: mostly cat, a little dog.Distillation: train a student with the same architecture, at the same temperature, on these soft labels.At temperature 1 the logits act \(T\) times larger. The softmax saturates, the gradient vanishes, FGSM fails.In the reported tests, C&W attacks the logits and breaks the distilled models. FGSM was misled by the gradient.
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016). Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP) (pp. 582–597). IEEE. https://doi.org/10.1109/SP.2016.41Carlini, N., & Wagner, D. (2017). Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP) (pp. 39–57). IEEE. https://doi.org/10.1109/SP.2017.49
Robust training · gradient regularisation

Gradient regularisation: flatten the loss around each input

Near an image, a smaller input gradient can mean a smaller loss increase—but only locally.

Gradient regularisation: flatten the loss around each inputSteep loss around xinputAfter penalising the slopeinputFlat at x, but curving upinput
\(x\)\(x+\epsilon\)\(\ell\)\(\epsilon\)\(\Delta\ell\approx\epsilon\times\text{slope}\)\(x\)\(x+\epsilon\)\(\ell\)\(\epsilon\)\(\Delta\ell\approx\epsilon\times\text{slope}\)\(x\)\(x+\epsilon\)\(\ell\)\(\epsilon\)\(\Delta\ell\) large again\(\displaystyle \max_{\|\delta\|_\infty\le\epsilon}\ell(x+\delta)\ \approx\ \ell(x)+\epsilon\,\big\|\nabla_x\ell(x)\big\|_1\)\(\displaystyle \min_\theta\ \underbrace{\ell\big(f_\theta(x),y\big)}_{\text{fit the label}}\;+\;\lambda\,\underbrace{\big\|\nabla_x \ell\big(f_\theta(x),y\big)\big\|_2^2}_{\text{keep the slope small}}\)Near \(x\) the loss is almost a straight line (dashed). The worst step of size \(\epsilon\) raises it by about \(\epsilon\) times the slope.Penalising the input slope can reduce nearby loss changes; it is not a proof for the whole box.Limitation: only the slope at \(x\) is penalised. The loss can be flat at \(x\) and still bend up sharply within \(\epsilon\).
Ross, A. S., & Doshi-Velez, F. (2018). Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11504
Obfuscated gradients

Obfuscated gradients: a false sense of security

A defence \(g\) sits before \(f\). In the tested cases, failed gradient attacks did not mean the models were safe.

Obfuscated gradients: a false sense of securityShatteredgradient✗e.g. rounding, JPEG, thermometer codesFix: BPDA†Thermometer encoding0%Input transformations0%Stochastice.g. random resizing, random pruningFix: EOT†Activation pruning0%Random resizing0%Vanishinge.g. purifying with a generative modelFix: ReparameterisePixelDefend9%Defense-GAN0%*
\(x\)\(g\)\(f\)\(\ell\)skip \(g\)The gradient through \(g\) is zeroOn the backward pass, skip \(g\): treat it as doing nothing.\(x\)\(g\)\(f\)\(\ell\)averageThe gradient changes on every queryTake gradients over many random draws, then average.\(x\)\(g\)\(h(z)\)\(f\)\(\ell\)The gradient fades through a long loopSwap the loop for a smooth function \(h\) and search over \(z\).* in principle; 55% measured in the paper.Athalye et al. (2018): 7 of 9 ICLR 2018 defences blocked the backward pass through \(g\), in one of three ways.Adaptive attacks can find usable directions through these three obstacles.In this comparison, most fell near zero; PGD training retained 47% under attack.
Athalye, A., Carlini, N., & Wagner, D. (2018). Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning (pp. 274–283). PMLR.Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations.
Adversarial training · why it works

Robust features vs non-robust features

Ilyas et al. (2019): adversarial examples are not bugs. They exploit real but fragile patterns in the data.

Robust features vs non-robust featuresRobust featuresshape, outline, earshumans see themNon-robust featurestiny pixel patternshumans cannot see thempredicts “cat”also predicts “cat”Normal trainingusesusesAn attack changes thetiny patterns:“cat” → “dog”Adversarial trainingusesignores
Fragile clues may matter lessfor this \(\epsilon\)-boxnot a promise for other attacksAn image holds two kinds of useful features. Both really do predict the label.Normal training uses both. An attack only needs to change the tiny patterns humans do not notice.Training against these small changes encourages the model to rely less on fragile patterns.
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019). Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 32.
Adversarial training · trade-offs

Robustness is not free

Robustness can cost accuracy, time, and data. The comparisons below are illustrative.

Robustness is not free1Lower clean accuracyNormal training95%Adversarial training87%2Slower trainingNormal trainingAdversarial training3Needs more dataNormal trainingRobust training4Robust overfittingtraintestbest
clean accuracy\(1\) pass\(K\) attack steps \(+\ 1\) updatepasses per training batchtraining data neededrobust accuracy over epochsAdversarial training can reduce clean accuracy when fragile features are useful.Each batch needs roughly \(K\) attack steps and one weight update; actual runtime depends on the implementation.Robust generalisation can need much more training data.Robust test accuracy may peak early while training accuracy keeps improving.
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., & Madry, A. (2019). Robustness may be at odds with accuracy. International Conference on Learning Representations.Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., & Madry, A. (2018). Adversarially robust generalization requires more data. Advances in Neural Information Processing Systems, 31.Rice, L., Wong, E., & Kolter, J. Z. (2020). Overfitting in adversarially robust deep learning. In Proceedings of the 37th International Conference on Machine Learning (pp. 8093–8104). PMLR.
Adversarial training · variants

Variants of adversarial training

Same min–max idea. Different losses, or cheaper ways to find the attack.

Change the loss
PGD-ATMadry et al., 2018
\(\displaystyle \min_\theta\ \max_{\|\delta\|\le\epsilon}\ \ell\big(f(x+\delta),y\big)\)

Train on the image found by the PGD search.

TRADESZhang et al., 2019
\(\displaystyle \min_\theta\ \ell\big(f(x),y\big)+\beta\,\max_{\|\delta\|\le\epsilon}\,\mathrm{KL}\big(f(x)\,\|\,f(x+\delta)\big)\)

Fit the clean image, and keep the output stable inside the box. \(\beta\) sets the trade-off.

MARTWang et al., 2020
\(\displaystyle \min_\theta\ \mathrm{BCE}\big(f(x+\delta),y\big)+\beta\,\mathrm{KL}\big(f(x)\,\|\,f(x+\delta)\big)\big(1-f_y(x)\big)\)

Like TRADES, but weight images more when the correct label has low probability.

Make it cheaper
Free ATShafahi et al., 2019
one backward pass updates both \(\theta\) and \(\delta\)

Replay batches and share gradient work to reduce cost.

Fast ATWong et al., 2020
\(\delta\sim\mathcal U[-\epsilon,\epsilon]\), then one FGSM step with \(\alpha=1.25\,\epsilon\)

Random-start FGSM can be faster; test whether it matches PGD training here.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations.Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., & Jordan, M. (2019). Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning (pp. 7472–7482). PMLR.Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., & Gu, Q. (2020). Improving adversarial robustness requires revisiting misclassified examples. International Conference on Learning Representations.Shafahi, A., Najibi, M., Ghiasi, A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., & Goldstein, T. (2019). Adversarial training for free! Advances in Neural Information Processing Systems, 32.Wong, E., Rice, L., & Kolter, J. Z. (2020). Fast is better than free: Revisiting adversarial training. International Conference on Learning Representations.
Test-time defences · guarantees

Certified robustness

Mathematically proven robustness: a guarantee, not just a test.

Certified robustnessAdversarial training: empiricalcatdogattacks we triednever checked: still flipsCertified robustness: provencatdog
\(x\)\(x\)\(\epsilon\)\(r\)Adversarial training is empirical, not theoretically proven: an attack we never tried may still work.A certified defence mathematically proves that every image within \(\epsilon\) of \(x\) is still a cat.The certified radius \(r\) is the largest ball we can prove. No attack inside it can ever succeed.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., & Kohli, P. (2018). On the effectiveness of interval bound propagation for training verifiably robust models [Preprint]. arXiv. https://arxiv.org/abs/1810.12715Cohen, J., Rosenfeld, E., & Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (pp. 1310–1320). PMLR.
Certified robustness · bounds

Bound the output of one neuron

If each pixel can only move by \(\epsilon=0.1\), how far can the score move?

Bound the output of one neuron+2−1+1scoreReLUReLU-0.500.51
each pixel \(\pm\,0.1\)\(0.2\)\(0.2\pm0.1\)\(0.8\)\(0.8\pm0.1\)\(0.5\)\(0.5\pm0.1\)before ReLU\(0.1\)\(-0.3\)\(0.5\)after ReLUbecomes 0\(0.5\)\(\text{score}=2\times0.2\;-\;1\times0.8\;+\;1\times0.5=0.1\)\(\text{lowest score}=2\times0.1-1\times0.9+1\times0.4=-0.3,\qquad\text{highest score}=2\times0.3-1\times0.7+1\times0.6=0.5\)\(\mathrm{ReLU}([-0.3,\ 0.5])=[\max(-0.3,0),\ \max(0.5,0)]=[0,\ 0.5]\)Take three pixels from the image. Their values are 0.2, 0.8 and 0.5.The neuron multiplies each pixel by its weight and adds them up. The score is 0.1.Now let each pixel move by up to ±0.1. In the worst case, the score can be anywhere from −0.3 to 0.5.ReLU turns every negative value into 0, so the range becomes 0 to 0.5.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., & Kohli, P. (2018). On the effectiveness of interval bound propagation for training verifiably robust models [Preprint]. arXiv. https://arxiv.org/abs/1810.12715
Certified robustness · interval bound propagation

Push the ranges through the whole network

Interval bound propagation (IBP): repeat the one-neuron rule, layer by layer.

Push the ranges through the whole network+1+1+1−1+1−1+1ReLUReLUReLUReLUcatdogpixelshiddenscores00.511.5catdogclean image: catCertified: always catCannot certifyNormal trainingclean imagecat1dog0IBP training addsworst case in the boxcat0.6dog0.2
\(h_1\)\(h_2\)\(0.5\)\(0.5\)\(1.0\)\(0\)\(1.0\)\(0\)\(1.0\)\(0\)\([0.4,\,0.6]\)\([0.4,\,0.6]\)\([0.8,\,1.2]\)\([-0.2,\,0.2]\)\([0.8,\,1.2]\)\([0,\,0.2]\)\([0.6,\,1.2]\)\([0,\,0.2]\)\([0.2,\,0.8]\)\([0.2,\,0.8]\)\([0.4,\,1.6]\)\([-0.6,\,0.6]\)\([0.4,\,1.6]\)\([0,\,0.6]\)\([-0.2,\,1.6]\)\([0,\,0.6]\)both pixels \(=0.5\)both pixels \(=0.5\), \(\ \epsilon=0.1\)both pixels \(=0.5\), \(\ \epsilon=0.3\)\(h_1=0.5+0.5=1.0,\ \ h_2=0.5-0.5=0,\qquad \text{cat}=\mathrm{ReLU}(h_1)-\mathrm{ReLU}(h_2)=1.0,\ \ \text{dog}=\mathrm{ReLU}(h_2)=0\)\(x_1,\,x_2\in[\,0.5-0.1,\ \ 0.5+0.1\,]=[0.4,\ 0.6]\)\(h_1=x_1+x_2\in[\,0.4+0.4,\ \ 0.6+0.6\,]=[0.8,\ 1.2],\qquad h_2=x_1-x_2\in[\,0.4-0.6,\ \ 0.6-0.4\,]=[-0.2,\ 0.2]\)\(\mathrm{ReLU}([0.8,\ 1.2])=[0.8,\ 1.2],\qquad \mathrm{ReLU}([-0.2,\ 0.2])=[\max(-0.2,0),\ \max(0.2,0)]=[0,\ 0.2]\)\(\text{cat}=\mathrm{ReLU}(h_1)-\mathrm{ReLU}(h_2)\in[\,0.8-0.2,\ \ 1.2-0\,]=[0.6,\ 1.2],\qquad \text{dog}=\mathrm{ReLU}(h_2)\in[0,\ 0.2]\)\(h_1\in[0.2+0.2,\,0.8+0.8]=[0.4,1.6],\ \ h_2\in[0.2-0.8,\,0.8-0.2]=[-0.6,0.6],\ \ \text{cat}\in[0.4-0.6,\,1.6-0]=[-0.2,1.6]\)\(\mathcal{L}=\kappa\cdot\underbrace{\ell(\text{clean scores},\,\text{cat})}_{\text{normal training}}\;+\;(1-\kappa)\cdot\underbrace{\ell(\text{worst-case scores},\,\text{cat})}_{\text{the worst case must also say cat}}\)A tiny network: two pixels in, a cat score and a dog score out. Both pixels are 0.5.Run the clean image through: cat gets 1.0, dog gets 0. The model says cat.Now an attacker may move each pixel by up to 0.1, so each pixel is somewhere in [0.4, 0.6].For each neuron, work out its smallest and largest possible value. A −1 weight flips which end we use.ReLU turns negative values into 0, so −0.2 becomes 0.Lowest cat score (0.6) beats highest dog score (0.2). Every image in the box is a cat.With a bigger budget \(\epsilon=0.3\), the ranges overlap. Dog might win, so IBP cannot prove anything.Normal training only asks the clean image to say cat. IBP training also asks the worst case to say cat.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., & Kohli, P. (2018). On the effectiveness of interval bound propagation for training verifiably robust models [Preprint]. arXiv. https://arxiv.org/abs/1810.12715
Certified robustness · interval bound propagation

Limitations of IBP

IBP is fast and simple. That speed comes at a price.

  • Ranges are too wide.It combines extremes that may not occur together. With more layers, these ranges may get too wide to certify the chosen budget.
  • Lower accuracy.At \(L_\infty\) budget \(8/255\), the cited CIFAR-10 IBP model got about half of the clean test images wrong.
  • Hard to train.In the cited training schedule, the budget grows gradually; a sudden increase can destabilise training.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., & Kohli, P. (2018). On the effectiveness of interval bound propagation for training verifiably robust models [Preprint]. arXiv. https://arxiv.org/abs/1810.12715
Certified robustness · randomised smoothing

Randomised smoothing: the basic idea

Add noise to the image many times. If most noisy copies are called cat, the answer is cat.

Randomised smoothing: the basic ideacatcatcatcatdogcatcatcatcatdogcatCount the votescatdog8 / 102 / 10
\(x\)\(f\)\(x\)\(+\eta\)\(f\) on each noisy copy\(g(x)=\text{cat}\)\(g(x)=\arg\max_{c}\ \Pr\big[f(x+\eta)=c\big],\qquad \eta\sim\mathcal{N}(0,\sigma^2 I)\)A normal model \(f\) gives one answer for one image.Add random Gaussian noise \(\eta\) many times, and ask \(f\) about every noisy copy.The smoothed model \(g\) returns the majority vote. We certify \(g\), not \(f\).
Cohen, J., Rosenfeld, E., & Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (pp. 1310–1320). PMLR.
Certified robustness · randomised smoothing

How far can an attacker move the image?

The vote does not flip at once. It needs a large enough move.

How far can an attacker move the image?dog90% catdogdog≥ 81% catdogdogdog≥ 68% catdogdogdogdogdog50% catcould flipBound: still cat
\(x\)\(x+\delta,\ \|\delta\|_2=0.1\)\(x+\delta,\ \|\delta\|_2=0.2\)\(x+\delta,\ \|\delta\|_2=0.32\)At \(x\), 9 out of 10 noisy copies are called cat (noise level \(\sigma=0.25\)).Any move smaller than 0.32 cannot flip the vote. Where does 0.32 come from?
Cohen, J., Rosenfeld, E., & Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (pp. 1310–1320). PMLR.
Certified robustness · randomised smoothing

Where the certified radius comes from

Gaussian noise around the input: blue area = cat votes, red area = dog votes.

Where the certified radius comes from
worst-case boundary\(x\)\(x+\delta\)\(x+\delta\)Gaussian PDFarea \(=\Phi\) (CDF)\(0.32\)90%10%74%26%50%50%Given\(\sigma=0.25\)noise std per pixel (pixels in [0, 1])we choose it; 0.25 is a common setting\(p_A=0.9\)we run the model \(f\) on 1000 noisy copies:900 say cat, 100 say dogWhere could the dog region be?Only 10% of the copies said dog, so dogcovers at most 10% of the bell. Worst case:the 10% tail on the attack side (red).Tail starts at \(\Phi^{-1}(0.9)\,\sigma=1.28\,\sigma\)\(=1.28\times0.25=0.32\)Perturbation \(\|\delta\|_2=0.16\)Distance to boundary: \(0.16=0.64\,\sigma\)Cat share: 74% \(\to\) still catPerturbation \(\|\delta\|_2=0.32\)\(x+\delta\) lies on the boundaryCat share: 50% \(\to\) tie\(\sigma=0.25\ \text{(noise level)},\qquad p_A=900/1000=0.9\ \text{(share of cat votes)}\)\(\text{cat share}=\Phi\!\Big(\dfrac{\text{distance}}{\sigma}\Big)=\Phi\!\Big(\dfrac{\text{distance}}{0.25}\Big)=0.9\ \Rightarrow\ \dfrac{\text{distance}}{0.25}=\Phi^{-1}(0.9)=1.28\ \Rightarrow\ \text{distance}=0.25\times1.28=0.32\)\(\text{cat share}=\Phi\!\Big(\dfrac{\text{distance left}}{\sigma}\Big)=\Phi\!\Big(\dfrac{0.32-0.16}{0.25}\Big)=\Phi\!\Big(\dfrac{0.16}{0.25}\Big)=\Phi(0.64)=0.74\)\(\text{cat share}=\Phi\!\Big(\dfrac{\text{distance left}}{\sigma}\Big)=\Phi\!\Big(\dfrac{0.32-0.32}{0.25}\Big)=\Phi(0)=0.5\ \Rightarrow\ R=0.32\)Two inputs: the noise level \(\sigma\), and the cat share \(p_A\) estimated by sampling.Worst case: dog is the red 10% tail. Its edge is the closest dog can be to \(x\).A perturbation of 0.16 shifts the noise with it. Cat keeps 74% of the votes.At 0.32 the vote is tied. Hence the certified radius is \(R=0.32\).
Cohen, J., Rosenfeld, E., & Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (pp. 1310–1320). PMLR.
Certified robustness · randomised smoothing

Randomised smoothing in practice

How we get the radius for one image.

Randomised smoothing in practiceCount the votescat90,000dog10,000cat50,100dog49,900ObservedCertifiedAbstainNoisy inputsCertified accuracy on ImageNet
image \(x\)100,000 noisy copiesmodel \(f\)cat share we counted\(\hat p_A=\dfrac{90{,}000}{100{,}000}=0.90\)\(n=100{,}000\) copies\(\alpha=0.001\): bound fails \(\le1/1000\)safety margin \(t=0.006\)safe guess \(p_A\ge0.894\)certified radius \(R=0.31\)safe guess of cat share\(p_A\ge0.501-t=0.495\)below \(0.5\): cat might loseCERTIFY abstains\(\sigma=0\)\(\sigma=0.25\)\(\sigma=0.50\)\(\sigma=1.00\)Noise is added to every pixel\(\hat p_A=\dfrac{\text{cat votes}}{n}=\dfrac{90{,}000}{100{,}000}=0.90\)\(\text{Hoeffding: }P\big(\hat p_A-p_A\ge t\big)\le e^{-2nt^2}=\alpha\ \Rightarrow\ t=\sqrt{\dfrac{\ln(1/\alpha)}{2n}}=\sqrt{\dfrac{6.91}{200{,}000}}\approx0.006\)\(p_A\ge0.501-0.006=0.495\lt0.5\ \Rightarrow\ \text{cat might lose}\ \Rightarrow\ \text{no certificate}\)\(\text{ImageNet, }\sigma=0.25:\quad67\%\text{ correct without attack; }\quad49\%\text{ certified correct at }L_2\text{ radius }0.5.\)Note: \(\hat p_A\) = the cat share we counted. \(p_A\) = the real cat share, which we can only estimate.The count can be high by chance; we will subtract a safety margin.\(\sigma=0.25\) is the Gaussian noise standard deviation chosen on the previous slide.Radius: we chose \(\sigma=0.25\) earlier; \(R=\sigma\Phi^{-1}(p_A)\) grows with \(\sigma\) at fixed vote share.Accuracy: larger \(\sigma\) makes the copies noisier, so the model is more likely to get them wrong.\(R\) = certified radius: any change to the image smaller than \(R\) cannot change the answer.Count 90,000 cat votes in 100,000 noisy copies: observed share \(\hat p_A=0.90\).Subtract the safety margin: \(0.90-0.006=0.894\). Then \(R=0.25\times\Phi^{-1}(0.894)=0.31\).If the safe cat share falls below 50%, the certification procedure abstains.For those certified images, no \(L_2\) change smaller than 0.5 can make the prediction wrong.
Cohen, J., Rosenfeld, E., & Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (pp. 1310–1320). PMLR.
Certified robustness · randomised smoothing

Limitations of randomised smoothing

Randomised smoothing gives a guarantee—but only within a certified radius.

  • Slow to certify.The paper uses 100,000 noisy copies per image. Predicting a label can use far fewer.
  • Only changes inside the radius.The guarantee uses \(L_2\) pixel distance, not visual similarity. A patch or rotation can fall outside it.
  • Lower accuracy.More noise can certify larger changes, but makes the images harder to recognise.
Cohen, J., Rosenfeld, E., & Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (pp. 1310–1320). PMLR.
Input preprocessing · methods

Input preprocessing

Same pipeline, different transformations. The images and scores are illustrative.

Input imageInput preprocessingClassifierPrediction
Input preprocessing pipeline adversarial cattransformnoneJPEG q = 75TV minimisationwhat the classifier seesClassifierP(cat)0.08“dog” ✗0.61“cat” ✓0.74“cat” ✓ Binarythresholding0.52“cat”
Without preprocessing, the small perturbation fools the classifier.JPEG compression removes fine detail.Total variation (TV) removes small pixel wiggles while preserving larger edges.Grey pixels below 0.5 become 0; the rest become 1. Values near 0.5 can still flip.

All these methods follow the same idea: transform first, then classify.

Guo, C., Rana, M., Cisse, M., & van der Maaten, L. (2018). Countering adversarial images using input transformations. International Conference on Learning Representations.Wang, Y., Zhang, W., Shen, T., Yu, H., & Wang, F.-Y. (2021). Binary thresholding defense against adversarial attacks. Neurocomputing, 445, 61–71. https://doi.org/10.1016/j.neucom.2021.03.036
Detection · adversarial examples

OOD detection for adversarial examples

Use anomaly and OOD detection methods to flag inputs that may be adversarial examples.

  • Flag potential adversarial examples.Use confidence, feature-space distance, or a detector trained on clean and attacked examples. Reject or review flagged inputs.
  • Limitation: false alarms.A stricter threshold catches more attacks, but also rejects more clean inputs.
  • Limitation: attacks can look normal.Adversarial inputs are not always OOD. An adaptive attack can fool both the classifier and the detector.
Carlini, N., & Wagner, D. (2017). Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (pp. 3–14). ACM. https://doi.org/10.1145/3128572.3140444Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations.Metzen, J. H., Genewein, T., Fischer, V., & Bischoff, B. (2017). On detecting adversarial perturbations. International Conference on Learning Representations.
Guardrails · input and output filtering

Guardrails: input and output filtering

Use a separate detector to screen the request and the draft reply.

Guardrails: input and output filteringUser promptbenign or harmfulInput guardscreens the requestMain LLMweights stay fixedOutput guardscreens the draftDelivered replyallowed answerRefuseharmful requestBlock / reviseharmful draftGuards can use rules or a safety classifier such as Llama Guard.
Like OOD detection: a check around the model. Fluent text can still be unsafe.Limitation: the guard can miss unsafe cases and can raise false alarms.
Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-Y., Goldblum, M., Saha, A., Geiping, J., & Goldstein, T. (2023). Baseline defenses for adversarial attacks against aligned language models [Preprint]. arXiv. https://arxiv.org/abs/2309.00614Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., & Khabsa, M. (2023). Llama Guard: LLM-based input-output safeguard for human-AI conversations [Preprint]. arXiv. https://arxiv.org/abs/2312.06674
Safety training · SFT and RLHF

Safety training: teach safer behaviour

Learn safe, helpful behaviour from example replies and human preferences.

Safety training: teach safer behaviour1 · SFTSupervised fine-tuningLearn from written examplesBenignrequestHelpfulreplyHarmfulrequestRefusal orsafe alternative2 · RankHumans rank repliesSafety and helpfulnessPromptUnsafe replyworseSafe + helpfulbetter3 · RLHFReinforcement learningfrom human feedback (RLHF)Reward modellearns human preferencesUpdate LLM for higher reward
Unlike a guardrail, safety training changes the LLM’s weights.Limitation: unsafe replies can remain, and harmless requests can be over-refused.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., … Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models [Preprint]. arXiv. https://arxiv.org/abs/2307.09288
Summary

Test-time defences: summary

  • Adversarial training.Train on attacked inputs with the correct labels. Robustness is tested against a chosen threat model.
  • Certified robustness.IBP and randomised smoothing can prove that a prediction stays unchanged inside a specified radius.
  • Input transformations and detectors.Preprocess suspicious inputs or reject them. Passing a detector is not a robustness guarantee.
  • LLM safety.Guardrails filter requests and replies. Safety training changes the model’s weights; neither removes every failure.

Next lectureDefences II: training-time attacks and the public good