The University of Melbourne

Generating Adversarial
Examples

COMP90073 Trustworthy Machine Learning Hanxun Huang, CIS Semester 2, 2026
Lecture map

Outline

01Why are linear classifiers useful for understanding attacks?
02How do we measure and constrain a perturbation?
03How do FGSM, PGD and MI-FGSM search with gradients?
Linear layers · foundation

Why start with a linear layer?

  • Linear layers are the building blocks of neural networks.
  • Classification head\(\color{#ff4756}{z}=\color{#000f46}{W}\color{#16834f}{h}+\color{#745b00}{b}\) gives class logits; \(\color{#ff4756}{p}=\color{#000f46}{\operatorname{softmax}}(\color{#ff4756}{z})\) gives class probabilities.
  • Attention projections\(\color{#ff4756}{Q}=\color{#16834f}{X}\color{#000f46}{W_Q}+\color{#745b00}{b_Q},\quad \color{#ff4756}{K}=\color{#16834f}{X}\color{#000f46}{W_K}+\color{#745b00}{b_K},\quad \color{#ff4756}{V}=\color{#16834f}{X}\color{#000f46}{W_V}+\color{#745b00}{b_V}\).
  • Convolution\(\color{#ff4756}{\operatorname{vec}(Y)}=\color{#000f46}{T(K)}\color{#16834f}{\operatorname{vec}(X)}+\color{#745b00}{b}\), where \(T(K)\) is Toeplitz in 1-D
    and block Toeplitz with Toeplitz blocks (BTTB) in 2-D.

outputweights / operatorinputbias

Linear layers · score geometry

A linear classifier partitions the input space

Linear score

For class \(i\):

\[g_i(x)=w_i^\top x+b_i\]
Prediction

The model selects the largest score:

\[\hat y=\arg\max_i g_i(x)\]
Decision boundary

Classes \(i\) and \(j\) meet when their scores are equal.

1

\[g_i(x)=g_j(x)\]

2

\[w_i^\top x+b_i=w_j^\top x+b_j\]

3

\[(w_i-w_j)^\top x+(b_i-b_j)=0\]

Class \(i\)
Class \(j\)
\(x_0\)
\(g_i(x)=g_j(x)\)
Linear layers · multiclass target

A multiclass target is an intersection of half-spaces

Boundary with class 1

Target class \(t\) ties class 1 on this line.

\[g_1(x)=g_t(x)\]
Boundary with class 2

Target class \(t\) ties class 2 on this line.

\[g_2(x)=g_t(x)\]
Target region

A \(K\)-class model contributes \(K-1\) comparisons—one per competing class.

\[\begin{aligned}g_1(x)\le g_t(x)&\iff (w_1-w_t)^\top x\le b_t-b_1\\g_2(x)\le g_t(x)&\iff (w_2-w_t)^\top x\le b_t-b_2\end{aligned}\]
\[ \underbrace{\begin{bmatrix} (w_1-w_t)^\top\\ (w_2-w_t)^\top \end{bmatrix}}_{A^\top\in\mathbb R^{2\times d}}x \ \le\ \underbrace{\begin{bmatrix} b_t-b_1\\ b_t-b_2 \end{bmatrix}}_{b\in\mathbb R^2} \]
\[ \underbrace{\begin{bmatrix} (w_1-w_t)^\top\\ \vdots\\ (w_{t-1}-w_t)^\top\\ (w_{t+1}-w_t)^\top\\ \vdots\\ (w_K-w_t)^\top \end{bmatrix}}_{A^\top\in\mathbb R^{(K-1)\times d}}x \ \le\ \underbrace{\begin{bmatrix} b_t-b_1\\ \vdots\\ b_t-b_{t-1}\\ b_t-b_{t+1}\\ \vdots\\ b_t-b_K \end{bmatrix}}_{b\in\mathbb R^{K-1}} \]
\(g_1(x)=g_t(x)\)
\(g_2(x)=g_t(x)\)
Target region \(\Omega_t\)
\(\Omega_t=\{x:A^\top x\le b\}\)
Linear layers · binary classification

Linear (Binary) Classifiers

1. Derive the boundary equation

Class \(t\) and class \(0\) meet where their scores are equal.

\[\begin{aligned} g_t(x)&=w_t^\top x+b_t,\qquad g_0(x)=w_0^\top x+b_0\\ m(x)&=g_t(x)-g_0(x)\\ g_t(x)=g_0(x)&\iff m(x)=0\\ m(x)&=(w_t-w_0)^\top x+(b_t-b_0)=z^\top x+z_0\\ z&=w_t-w_0,\qquad z_0=b_t-b_0 \end{aligned}\]
2. Introducing a Lagrange multiplier
Boundary
\[m(x)=z^\top x+z_0=0\]
Problem
\[\min_x\;\underbrace{\tfrac12\|x-x_0\|_2^2}_{f(x)}\quad\text{s.t.}\quad m(x)=0\]
Lagrangian
\[\mathcal L=f(x)+\lambda m(x)=\tfrac12\|x-x_0\|_2^2+\lambda(z^\top x+z_0)\]

To find the analytic minimum, we need to find the partial derivatives and set them to \(0\).

Differentiate \(x\)
\[\begin{aligned}\nabla_x\mathcal L&=\nabla_x f(x)+\lambda\nabla_x m(x)\\&=(x-x_0)+\lambda z=0\end{aligned}\]
Differentiate \(\lambda\)
\[\frac{\partial\mathcal L}{\partial\lambda}=m(x)=z^\top x+z_0=0\]
Class \(0\): \(m(x)<0\)
Target \(t\): \(m(x)\ge0\)
distance from \(x_0\)
\(x_0\)
Linear layers · analytic attack

Solving the Lagrange equations

Goal: solve for \(\lambda\), then recover \(x^*\).

From previous slide
\[\nabla_x\mathcal L=(x-x_0)+\lambda z=0,\qquad \frac{\partial\mathcal L}{\partial\lambda}=z^\top x+z_0=0\]
Left-multiply by \(z^\top\)
\[\begin{aligned}z^\top(x-x_0+\lambda z)&=z^\top0=0\\z^\top x-z^\top x_0+\lambda\|z\|_2^2&=0\\z^\top x&=z^\top x_0-\lambda\|z\|_2^2\end{aligned}\]
Substitute the constraint
\[\begin{aligned}z^\top x&=z^\top x_0-\lambda\|z\|_2^2\\z^\top x&=-z_0\\z^\top x_0-\lambda\|z\|_2^2&=-z_0\\\lambda\|z\|_2^2&=z^\top x_0+z_0\end{aligned}\]
Divide by \(\|z\|_2^2\)
\[\begin{aligned}\lambda^*\|z\|_2^2&=z^\top x_0+z_0\\\lambda^*&=(z^\top x_0+z_0)\frac{1}{\|z\|_2^2}\\&=\frac{z^\top x_0+z_0}{\|z\|_2^2}\end{aligned}\]
Recover \(x^*\)
\[x^*=x_0-\lambda^*z=x_0-\frac{z^\top x_0+z_0}{\|z\|_2^2}\,z\]
Linear binary classifier · result

One projection gives the exact attack

Class \(0\)
Target \(t\)
\(m(x)=0\)
\(x_0\)
\(x^*\)
Shortest direction

Move along the boundary normal \(z\).

\[r^*=x^*-x_0=-\lambda^*z\]
Minimum distance

The margin determines how far \(x_0\) is from the boundary.

\[\|r^*\|_2=\frac{|z^\top x_0+z_0|}{\|z\|_2}\]
Closed-form result
\[x^*=x_0-\frac{z^\top x_0+z_0}{\|z\|_2^2}z\]

One projection. Exact for a binary linear classifier.

Optimization · conceptual idea

Use gradient descent to find adversarial examples

Define an attack objective

Choose a differentiable \(F\) whose minimum corresponds to attack success.

\[x_{\mathrm{adv}}^*\in\arg\min_{x_{\mathrm{adv}}}F(x_{\mathrm{adv}})\]
Take gradient descent steps

Recompute the local descent direction after every update.

\[\begin{aligned}d^{(k)}&=\nabla_xF(x_{\mathrm{adv}}^{(k)})\\x_{\mathrm{adv}}^{(k+1)}&=x_{\mathrm{adv}}^{(k)}-\gamma_kd^{(k)}\end{aligned}\]
Stop when the attack criterion is met

For an untargeted attack, stop at the first iterate whose predicted class changes.

\[\begin{aligned}K&=\min\{k\ge1:\hat y(x_{\mathrm{adv}}^{(k)})\ne\hat y(x_{\mathrm{adv}}^{(0)})\}\\x_{\mathrm{adv}}&=x_{\mathrm{adv}}^{(K)}\end{aligned}\]

Gradient → update → check → repeat

\(x_{\mathrm{adv}}^{(0)}\)
\(-\nabla_xF(x_{\mathrm{adv}}^{(k)})\)
\(x_{\mathrm{adv}}=x_{\mathrm{adv}}^{(K)}\)
Attack constraints · foundations

What Are Norms?

\(\delta\) is a perturbation vector; \(\varepsilon\) is its allowed radius under the chosen norm.

\[\begin{aligned}\|\delta\|_p&=\left(\sum_{i=1}^{d}|\delta_i|^p\right)^{1/p}\\\|\delta\|_p&\le\varepsilon\qquad\text{allowed perturbation}\\\|\delta\|_p&=\varepsilon\qquad\text{boundary at radius }\varepsilon\end{aligned}\]
\(\ell_1\)
\[\|\delta\|_1=\sum_i|\delta_i|\]

Total absolute change

\(\ell_2\)
\[\|\delta\|_2=\sqrt{\sum_i\delta_i^2}\]

Euclidean distance

\(\ell_\infty\)
\[\|\delta\|_\infty=\max_i|\delta_i|\]

Largest coordinate change

\(\delta_1\)
\(\delta_2\)
\(+\varepsilon\)
\(-\varepsilon\)
\(+\varepsilon\)
\(-\varepsilon\)
\(0\)
\(\ell_1\) · radius \(\varepsilon\) · diamond \(\ell_2\) · radius \(\varepsilon\) · circle \(\ell_\infty\) · radius \(\varepsilon\) · square
Axis intercepts are \(\pm\varepsilon\); every boundary satisfies \(\|\delta\|_p=\varepsilon\).
Attack constraint · norm choice

How do we measure perturbation strength?

\[\delta=x_{\mathrm{adv}}-x_0,\qquad \|\delta\|_p\le\varepsilon\]

\[\|\delta\|_1\le\varepsilon\]
\[\|\delta\|_2\le\varepsilon\]
\[\|\delta\|_\infty\le\varepsilon,\qquad \varepsilon=\frac{N}{255}\]
Original photograph of three cats
Original \(x_0\)
Perturbed \(x_{\mathrm{adv}}\)
Noise is hard to see
subtlefull noise
Attack constraints · semantic preservation

Why constrain the perturbation?

01Start with the original input

The semantic label is 1, and the model predicts 1.

Grayscale handwritten digit 1
Semantic label: 1 · Model: 1
→
02Keep the change small

The semantic label stays 1. If the model changes its prediction, this is a valid attack.

Handwritten digit 1 after a small pixel perturbation
Semantic label: 1 · Model: changed
→
03Do not change the meaning

If the image now represents a 7, its semantic label has changed. This is not an attack on the original 1.

Grayscale handwritten digit 7
Semantic label: 7 · Not an attack
Gradient directions · comparison

Comparison of Three Types
of Gradient-related Vectors

Gradient vectors\(\nabla f\)
\[\|v\|_2=\|\nabla f\|_2\quad\text{varies}\]

True direction and strength of steepest ascent; magnitude increases away from the origin.

Sign of gradient vectors\(\operatorname{sign}(\nabla f)\)
\[v_i\in\{-1,0,+1\}\]

Each component is \(-1\), \(0\), or \(+1\); direction signs remain, magnitude is discarded.

Normalized gradient vectors\(\nabla f/\|\nabla f\|_2\)
\[\|v\|_2=1\quad\text{everywhere}\]

Same direction as the gradient, but unit length everywhere.

Practical attacks · high-level idea

Constructing An Adversarial Example

1. Choose a perturbation

Add \(\delta\) to the original input.

\[x_{\mathrm{adv}}=x_0+\delta\]
2. Increase the loss

Push the model away from the correct class.

\[\max_{\delta}\;\mathcal L(f_\theta(x_0+\delta),y)\]
3. Keep the change small
\[\max_{\delta}\;\mathcal L(f_\theta(x_0+\delta),y)\quad\text{s.t.}\quad\|\delta\|_p\le\varepsilon,\;\;x_0+\delta\in[0,1]^d\]

Keep pixels valid; stop when the prediction changes.

Candidate \(x_{\mathrm{adv}}=x_0+\delta\)

Add \(\delta\)

↓ adjust \(\delta\)
Optimization \(\mathcal L(f_\theta(x_{\mathrm{adv}}),y)\uparrow\)

Increase the loss

↓ check the result
Success \(\hat y(x_{\mathrm{adv}})\ne y\)

Prediction changes; \(\delta\) stays small

Practical attacks · one-step method

Fast Gradient Sign Method (FGSM)

1. Evaluate the input gradient

Differentiate the loss with respect to \(x\) at the original input \(x_0\).

\[\nabla_x\mathcal L(x_0,y)\]
2. Take the sign for an \(\ell_\infty\) step

Each coordinate takes its largest allowed increase or decrease; \(\varepsilon\) sets that maximum change.

\[\delta_{\mathrm{FGSM}}=\varepsilon\,\operatorname{sign}\!\left(\nabla_x\mathcal L(x_0,y)\right)\]
3. Update the input once
\[x_{\mathrm{adv}}=\operatorname{clip}_{[0,1]}\!\left(x_0+\varepsilon\,\operatorname{sign}\!\left(\nabla_x\mathcal L(x_0,y)\right)\right)\]

Fast: one backward pass and one update.

4. Adapt the direction for an \(\ell_2\) budget

Replace the sign vector with the unit-normalized gradient.

\[x_{\mathrm{adv}}=x_0+\varepsilon\,\frac{\nabla_x\mathcal L(x_0,y)}{\|\nabla_x\mathcal L(x_0,y)\|_2}\]
\(\|\delta\|_\infty\le\varepsilon\)
\(x_0\)
\(\nabla_x\mathcal L(x_0,y)\)
\(\delta_{\mathrm{FGSM}}\)
\(x_{\mathrm{adv}}\)
\(\ell_2\): normalized gradient
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. International Conference on Learning Representations.
Practical attacks · one-step method

Why could one FGSM step fail?

Correct local direction

At \(x_0\), the sign gradient points toward a nearby different-class region.

One endpoint update

FGSM jumps to the edge of the \(\ell_\infty\) budget without checking the points in between.

Overshoot

The step crosses the curved region, but its endpoint is back in the original class.

\(\|\delta\|_\infty\le\varepsilon\)
different class
\(x_0\)
correct local sign
FGSM endpoint · \(\hat y=y\)
small steps · recompute
stop · \(\hat y\ne y\)
Practical attacks · iterative method

Projected Gradient Descent (PGD)

1. Start inside the budget

Start at \(x_0\), or add a random perturbation inside the same \(\ell_\infty\) budget.

\[x_{\mathrm{adv}}^{(0)}=x_0+\delta^{(0)},\qquad \|\delta^{(0)}\|_\infty\le\varepsilon\]
2. Take a small sign-gradient step

Recompute the gradient at the current point.

\[\widetilde x_{\mathrm{adv}}^{(k+1)}=x_{\mathrm{adv}}^{(k)}+\alpha\,\operatorname{sign}\!\left(\nabla_x\mathcal L(x_{\mathrm{adv}}^{(k)},y)\right)\]
3. Project back into the budget

Keep the point within \(\varepsilon\) of \(x_0\) and inside the valid pixel range.

\[x_{\mathrm{adv}}^{(k+1)}=\Pi_{\mathcal C}\!\left(\widetilde x_{\mathrm{adv}}^{(k+1)}\right),\qquad \mathcal C=\{z:\|z-x_0\|_\infty\le\varepsilon\}\cap[0,1]^d\]

Repeat → recompute → project

\(\|\delta\|_\infty\le\varepsilon\)
\(x_{\mathrm{adv}}^{(0)}\)
project back
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations.
Practical attacks · momentum

Momentum Iterative Fast Gradient Sign Method (MI-FGSM)

1. Initialize the attack

Use \(T\) steps. Each step has size \(\alpha=\varepsilon/T\).

\[x_{\mathrm{adv}}^{(0)}=x_0,\qquad g^{(0)}=0,\qquad \alpha=\frac{\varepsilon}{T}\]
2. Accumulate normalized gradients

\(\mu\) keeps part of the previous accumulated direction.

\[g^{(k+1)}=\mu g^{(k)}+\frac{\nabla_x\mathcal L(x_{\mathrm{adv}}^{(k)},y)}{\|\nabla_x\mathcal L(x_{\mathrm{adv}}^{(k)},y)\|_1}\]
3. Update with the accumulated sign

Use \(\operatorname{sign}(g^{(k+1)})\), not only the current gradient.

\[x_{\mathrm{adv}}^{(k+1)}=x_{\mathrm{adv}}^{(k)}+\alpha\,\operatorname{sign}(g^{(k+1)}),\qquad k=0,\ldots,T-1\]

\(x_{\mathrm{adv}}=x_{\mathrm{adv}}^{(T)}\), with \(\|x_{\mathrm{adv}}-x_0\|_\infty\le\varepsilon\)

without momentum
with momentum
oscillating updates
stable direction
Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., & Li, J. (2018). Boosting adversarial attacks with momentum. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9185–9193.
Summary

What have we learned?

  • Linear geometryFind the closest decision-boundary point with a closed-form projection.
  • Norms and budgetsL1, L2 and L-infinity define different perturbation budgets; pixels must stay valid.
  • Sign-step attacksFGSM takes 1 signed step. PGD repeats the step and projects back into the budget.
  • MomentumMI-FGSM accumulates a direction instead of following only the latest gradient.

Next lectureWhite-Box Attacks and Automatic Differentiation