Continuous \(x_{\mathrm{adv}}\in\mathbb R^d\)
A small numeric step is still a valid input.
\[x_{\mathrm{adv}}^{(k+1)}=\Pi_{\mathcal C}\!\left(x_{\mathrm{adv}}^{(k)}+\alpha d^{(k)}\right)\]
small valid move
Generating Adversarial
Examples (Part II)
COMP90073Trustworthy Machine LearningHanxun Huang, CISSemester 2, 2026
Outline
01C&W, GCG and AdvGAN
02Where do the derivatives used by these attacks come from?
03How do we implement them with automatic differentiation?
Carlini & Wagner (C&W) Attack
C&W replaces the hard class-change constraint with a logit penalty.
01 · Distance
\[\delta=x_{\mathrm{adv}}-x_0,\qquad d=\|\delta\|_2^2\]
Keep the perturbation small.
02 · Logit margin
\[m_y(x_{\mathrm{adv}})=Z_y(x_{\mathrm{adv}})-\max_{j\in\mathcal K\setminus\{y\}}Z_j(x_{\mathrm{adv}})\]
A non-positive margin means success.
03 · C&W objective
\[\begin{aligned}F(x_{\mathrm{adv}})&=\|\delta\|_2^2+c\max\!\left(m_y(x_{\mathrm{adv}}),0\right),\\x_{\mathrm{adv}}^*&\in\arg\min_{x_{\mathrm{adv}}}F(x_{\mathrm{adv}})\end{aligned}\]
\(c\) balances the two goals.
Turn C&W into an iterative method
The C&W objective is unconstrained, so we can minimize it with gradient descent.
01 · Objective
\[F(x_{\mathrm{adv}})=\|x_{\mathrm{adv}}-x_0\|_2^2+c\max\!\left(m_y(x_{\mathrm{adv}}),0\right)\]
One scalar combines distance and attack success.
02 · Descent update
\[x_{\mathrm{adv}}^{(k+1)}=x_{\mathrm{adv}}^{(k)}-\gamma\nabla_xF(x_{\mathrm{adv}}^{(k)})\]
Compute the gradient at the current \(x_{\mathrm{adv}}^{(k)}\), then take one step.
03 · Iterate
\[x_{\mathrm{adv}}^{(0)}=x_0,\qquad x_{\mathrm{adv}}=x_{\mathrm{adv}}^{(T)}\]
Recompute the gradient after every update.
C&W gradient, step by step
Evaluate every term at the current \(x_{\mathrm{adv}}\). Differentiate the two terms, then add them.
01 · Distance term
\[\begin{aligned}D(x_{\mathrm{adv}})&=\|x_{\mathrm{adv}}-x_0\|_2^2=\|\delta\|_2^2,\quad \delta=x_{\mathrm{adv}}-x_0,\\\nabla_xD(x_{\mathrm{adv}})&=2(x_{\mathrm{adv}}-x_0)=2\delta\end{aligned}\]
The derivative of \(\|u\|_2^2\) is \(2u\).
02 · Attack term
\[\begin{aligned}j^*&=\arg\max_{j\in\mathcal K\setminus\{y\}}Z_j && \text{best alternative}\\m_y&=Z_y-Z_{j^*} && \text{score gap}\\A&=c\max(m_y,0) && \text{penalty while }y\text{ wins}\\\nabla_xA&=c\,\mathbf 1[m_y>0]\!\left(\nabla_xZ_y-\nabla_xZ_{j^*}\right) && \text{direction to change class}\end{aligned}\]
03 · Combine and update
\[\begin{aligned}\nabla_xF&=\nabla_xD+\nabla_xA\\&=2\delta+c\,\mathbf 1[m_y>0]\!\left(\nabla_xZ_y-\nabla_xZ_{j^*}\right)\\x_{\mathrm{adv}}^{(k+1)}&=x_{\mathrm{adv}}^{(k)}-\gamma\nabla_xF\end{aligned}\]
Add the two gradients.
Then take one descent step.
From continuous inputs to discrete tokens
Every attack so far updates \(x_{\mathrm{adv}}\) in a continuous space. An LLM prompt becomes a sequence of discrete token IDs.
Discrete token IDs \(x=(x_1,\ldots,x_n)\)
Each token \(s_i\) has an ID \(x_i\), and an embedding \(e_i=E[x_i]\).
TheID 464answerID 3280isID 318safeID 3338
Problem: token IDs are discrete labels, so a gradient step cannot directly produce the next valid token ID.
Discrete Token IDs into embedding vectors
Embedding lookup maps each Token ID to a continuous vector.
\[\mathcal L_{\mathrm{CE}}=-\sum_{j=1}^{m}\log p_\theta\!\left(r_j\mid x_{\mathrm{prompt}},r_{<j}\right)\]
Greedy Coordinate Gradient (GCG)
GCG edits a token suffix to make a chosen reply more likely.
Edit the suffixKeep the original query fixed.
\[x_{\mathrm{adv}}=x_{\mathrm{prompt}}\,\Vert\,x_{\mathrm{suffix}}\]
Query + editable suffixHow to build a bomb+! ! ! !
Pick a target replyChoose the opening the model should produce.
\[r_{1:H}^*=\operatorname{tokens}\!\left(\text{“Sure, here is”}\right)\]
Target Token IDs1088911994318
Lower the target lossTry valid suffix tokens and keep the best result.
Target-prefix loss
\[\mathcal L_{\mathrm{CE}}(x_{\mathrm{adv}})=-\sum_{j=1}^{H}\log p_\theta\!\left(r_j^*\mid x_{\mathrm{adv}},r_{<j}^*\right)\]
Search over suffix tokens
\[x_{\mathrm{suffix}}^*\in\arg\min_{x_{\mathrm{suffix}}\in\mathcal V^s}\mathcal L_{\mathrm{CE}}\!\left(x_{\mathrm{prompt}}\Vert x_{\mathrm{suffix}}\right)\]
Lower loss → chosen reply is more likely.
Score every suffix position
The loss first reaches each embedding \(e_i\), then the lookup maps it to vocabulary scores for the suffix.
Target-prefix cross-entropy
\[\mathcal L_{\mathrm{CE}}(x_{\mathrm{adv}})=-\sum_{j=1}^{H}\log p_\theta\!\left(r_j^*\mid x_{\mathrm{adv}},r_{<j}^*\right)\]
Full context: prompt + suffix + earlier target tokens
One-hot token indicatorExample: Token ID 2 → \([0,\,0,\,1,\,0,\,\ldots]\)
\[\mathbf o_i=\operatorname{onehot}(x_i)\in\{0,1\}^{|\mathcal V|},\qquad e_i=E^\top\mathbf o_i\]
\[g_i=\nabla_{\mathbf o_i}\mathcal L_{\mathrm{CE}}\in\mathbb R^{|\mathcal V|}\]
Token ID \(v\)
04643338⋯\(g_{i_1}[v]\)+0.4−2.8−2.3⋯Token ID \(v\)
03280318⋯\(g_{i_2}[v]\)+0.2−2.6−2.1⋯Token ID \(v\)
045181657⋯\(g_{i_3}[v]\)+0.5−2.4−2.0⋯Token ID \(v\)
01330⋯\(g_{i_4}[v]\)+0.1−2.2−1.9⋯Shortlist tokens for each suffix position
Keep the query fixed. Use each suffix gradient to shortlist replacement Token IDs.
One shortlist per suffixKeep the fixed query unchanged.
\[\mathcal X_i=\operatorname{Top\text{-}k}(-g_i),\qquad i\in\mathcal I\]
Illustrative \(k=3\)
Token ID \(v\)\(g_{i_1}[v]\)
0+0.4464−2.83338−2.32450−1.9Token ID \(v\)\(g_{i_2}[v]\)
0+0.23280−2.6318−2.1262−1.8Token ID \(v\)\(g_{i_3}[v]\)
0+0.54518−2.41657−2.0584−1.7Token ID \(v\)\(g_{i_4}[v]\)
0+0.113−2.230−1.911−1.6Highlight the most negative \(g_i[v]\) entries. Return their Token IDs.
Build a batch of one-token replacements
Start each candidate from the same suffix. Change one position using its shortlist.
\(\mathcal X_{i_1}=\{464,3338,2450\}\)
\(\mathcal X_{i_2}=\{3280,318,262\}\)
\(\mathcal X_{i_3}=\{4518,1657,584\}\)
\(\mathcal X_{i_4}=\{13,30,11\}\)
Copy → choose position → replace one Token ID
Each row is an independent candidate, not the next step of the previous row.
CandidateReplacementFixed \(x_{\mathrm{prompt}}\)\(i_1\)\(i_2\)\(i_3\)\(i_4\)
\(b=1\)\(i_1\): 0 → 464
How to build a bomb
464The0!0!0!\(b=2\)\(i_2\): 0 → 3280
How to build a bomb
0!3280answer0!0!\(b=3\)\(i_3\): 0 → 1657
How to build a bomb
0!0!1657light0!\(b=4\)\(i_4\): 0 → 13
How to build a bomb
0!0!0!13.Stack the complete inputs\(\widetilde x_{\mathrm{adv}}^{(b)}=x_{\mathrm{prompt}}\Vert\widetilde x_{\mathrm{suffix}}^{(b)}\)
→
One model batch · \(B=4\) shownEvaluate each candidate with the same target-prefix CE loss.
Evaluate the batch and keep the best
Run all candidates together. Compare their losses.
Cross-entropy for sample b
\[\mathcal L_{\mathrm{CE}}^{(b)}=-\sum_{j=1}^{H}\log p_\theta\!\left(r_j^*\mid\widetilde x_{\mathrm{adv}}^{(b)},r_{<j}^*\right)\]
One loss per sample.
\(\mathcal L_{\mathrm{CE}}^{(1)}\)
2.41\(\mathcal L_{\mathrm{CE}}^{(2)}\)
1.73\(\mathcal L_{\mathrm{CE}}^{(3)}\)
2.08\(\mathcal L_{\mathrm{CE}}^{(4)}\)
1.92Keep sample 2 · lowest loss 1.73Use this candidate in the next iteration.
Repeat with the selected candidate
Keep the query fixed. Update the suffix, then start the next iteration.
Score each suffix position.
One shortlist per position.
One token replacement per candidate.
Keep the lowest-loss candidate.
Train a model to generate the perturbation
Train \(G_\phi\) across many images. Keep the target classifier \(f_\theta\) fixed.
\(x_1,x_2,x_3,\ldots\)
\(G_\phi(x)=\delta\)
\(+\)
original input \(x\) · skip connection
\(x_{\mathrm{adv}}=x+\delta\)
\(f_\theta\to\mathcal L_{\mathrm{attack}}\)
\(D\to\mathcal L_{\mathrm{GAN}}\)
\(D\) also receives real training images
Backpropagate loss · update \(\phi\) · repeat over batches
\(\mathbb E_x\,\ell(f_\theta(x+G_\phi(x)),t)\)
Push prediction toward target \(t\).\(\min_G\max_D\;\mathcal L_{\mathrm{GAN}}\)
Keep perturbed images realistic.\(\mathbb E_x\max(0,\|G_\phi(x)\|_2-c)\)
Discourage large perturbations.\(\mathcal L=\mathcal L_{\mathrm{attack}}+\alpha\mathcal L_{\mathrm{GAN}}+\beta\mathcal L_{\mathrm{size}}\) · Learn the generator, not a separate perturbation for each image.
Xiao, C., Li, B., Zhu, J.-Y., He, W., Liu, M., & Song, D. (2018). Generating adversarial examples with adversarial networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (pp. 3905–3911). International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2018/543Same trained generator, different new inputs
Freeze \(G_\phi\). For each new \(x_i\), generate its own \(\delta_i\) and construct \(x_{\mathrm{adv},i}=x_i+\delta_i\).
\(G_\phi\)
same \(\phi\) for every input
\(x_1\)
\(\delta_1=G_\phi(x_1)\) · add \(x_1\)
\(x_{\mathrm{adv},1}=x_1+\delta_1\)
\(x_2\)
\(\delta_2=G_\phi(x_2)\) · add \(x_2\)
\(x_{\mathrm{adv},2}=x_2+\delta_2\)
\(x_3\)
\(\delta_3=G_\phi(x_3)\) · add \(x_3\)
\(x_{\mathrm{adv},3}=x_3+\delta_3\)
Generalise across inputs: same \(G_\phi\), different \(\delta_i\). Not a single universal perturbation.
Xiao, C., Li, B., Zhu, J.-Y., He, W., Liu, M., & Song, D. (2018). Generating adversarial examples with adversarial networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (pp. 3905–3911). International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2018/543What Do All These Techniques Share?
FGSM · PGD · MI-FGSM · C&W · GCG · AdvGAN
- Choose a loss. The loss describes the behavior the attack wants.
- Compute a gradient. Gradients guide input updates or generator training.
-
Change the input. Keep the target model fixed.
- Images: change pixels, \(x+\delta\).
- Text: replace suffix Tokens, \(x_{\mathrm{prompt}}\Vert x_{\mathrm{suffix}}\).
- Computing model gradients requires white-box access. AdvGAN uses gradients during training—not during generation.
Where Do Derivatives Come From?
\(\hat y=f_\theta(x)\qquad \mathcal L=\mathcal L(f_\theta(x),y)\)
- Training updates \(\theta\). Keep \(x\) and \(y\) fixed: \(\theta\leftarrow\theta-\eta\nabla_\theta\mathcal L\).
- An attack updates \(x\). Keep \(\theta\) fixed: \(x_{\mathrm{adv}}\leftarrow x_{\mathrm{adv}}+\alpha\nabla_x\mathcal L\).
- For \(\hat y=wx+b\), training differentiates \(w\) and \(b\). Gradients: \(\frac{\partial\mathcal L}{\partial w}=\frac{\partial\mathcal L}{\partial\hat y}x\), \(\frac{\partial\mathcal L}{\partial b}=\frac{\partial\mathcal L}{\partial\hat y}\). Updates: \(w\leftarrow w-\eta\frac{\partial\mathcal L}{\partial\hat y}x\), \(b\leftarrow b-\eta\frac{\partial\mathcal L}{\partial\hat y}\).
- For the same layer, an attack differentiates \(x\). Gradient: \(\frac{\partial\mathcal L}{\partial x}=\frac{\partial\mathcal L}{\partial\hat y}w\). Update: \(x_{\mathrm{adv}}\leftarrow x_{\mathrm{adv}}+\alpha\frac{\partial\mathcal L}{\partial\hat y}w\).
The core pattern: upstream × local
Each node receives an upstream gradient and computes:
\[\text{downstream gradient}
=
\underbrace{\frac{\partial\mathcal L}{\partial\,\text{output}}}_{\text{upstream}}
\times
\underbrace{\frac{\partial\,\text{output}}{\partial\,\text{input}}}_{\text{local}}\]
inputs \(\;x,y\)
\[f\]
output \(\;z\)
\[\frac{\partial\mathcal L}{\partial z}\quad\text{upstream}\]
\[\frac{\partial\mathcal L}{\partial x}\quad\text{downstream}\]
Why this matters: each node needs only its own operation and the upstream gradient. This makes backpropagation modular and scalable.
Patterns in gradient flow
Let \(g_z\equiv\frac{\partial\mathcal L}{\partial z}\) be the upstream gradient arriving from the loss.
Add gatecopies the upstream gradient
\[g_z\]
\[g_a\]
\[g_b\]
\[a\]
\[b\]
\[+\]
\[z=a+b\]
\[\mathcal L\]
Local\(\displaystyle\frac{\partial z}{\partial a}=1,\qquad\frac{\partial z}{\partial b}=1\)
\[g_a\equiv\frac{\partial\mathcal L}{\partial a}=g_z,\qquad g_b\equiv\frac{\partial\mathcal L}{\partial b}=g_z\]
Multiply gateswaps and scales
\[g_z\]
\[g_zb\]
\[g_za\]
\[a\]
\[b\]
\[\times\]
\[z=ab\]
\[\mathcal L\]
Local\(\displaystyle\frac{\partial z}{\partial a}=b,\qquad\frac{\partial z}{\partial b}=a\)
\[g_a=g_z\,b,\qquad g_b=g_z\,a\]
Max gateroutes to the winner
\[g_z\]
\[g_z\mathbf 1[a>b]\]
\[g_z\mathbf 1[b>a]\]
\[a\]
\[b\]
max
\[z=\max(a,b)\]
\[\mathcal L\]
Local\(\displaystyle\frac{\partial z}{\partial a}=\mathbf 1[a>b],\qquad\frac{\partial z}{\partial b}=\mathbf 1[b>a]\)
\[g_a=g_z\mathbf 1[a>b],\qquad g_b=g_z\mathbf 1[b>a]\]
At \(a=b\), max is mathematically not differentiable. The tie-breaking or subgradient rule depends on the implementation.
Copy or branchadds incoming gradients
\[g_u\]
\[g_v\]
\[g_u+g_v\]
\[a\]
\[u=a\]
\[v=a\]
\[\mathcal L\]
Local\(\displaystyle\frac{\partial u}{\partial a}=1,\qquad\frac{\partial v}{\partial a}=1\)
\[g_a=g_u\frac{\partial u}{\partial a}+g_v\frac{\partial v}{\partial a}=g_u+g_v\]
Forward values, backward gradients
\[z=w_3\max(0,x_1w_1+x_2w_2)+b,\qquad v=\sigma(z),\qquad \mathcal L_{\mathrm{BCE}}=-y\log v-(1-y)\log(1-v),\quad y\in\{0,1\}\]
input\[x_1\]
weight\[w_1\]
multiply\[\times\]
value\[u_1\]
input\[x_2\]
weight\[w_2\]
multiply\[\times\]
value\[u_2\]
add\[+\]
value\[a\]
threshold\[0\]
max node\[\max\]
value\[h\]
weight\[w_3\]
multiply\[\times\]
value\[s\]
bias\[b\]
add\[+\]
logit\[z\]
target\[y\]
sigmoid\[v=\frac{1}{1+e^{-z}}\]
binary cross entropy\[\mathcal L_{\mathrm{BCE}}\]
\[g_v\]
\[g_z\]
\[g_s\]
\[g_b\]
\[g_h\]
\[g_{w_3}\]
\[g_a\]
\[g_{u_1}\]
\[g_{w_1}\]
\[g_{x_1}\]
\[g_{u_2}\]
\[g_{w_2}\]
\[g_{x_2}\]
Multiply \(w_1\)\(u_1=x_1w_1\)
Multiply \(w_2\)\(u_2=x_2w_2\)
Add inputs\(a=u_1+u_2\)
Max / ReLU\(h=\max(0,a)\)
Multiply \(w_3\)\(s=hw_3\)
Add \(b\)\(z=s+b\)
Sigmoid\(v=\frac{1}{1+e^{-z}}\)
BCE\(\mathcal L_{\mathrm{BCE}}(v,y)\)
NodeLocal ruleBackward: upstream × local
BCE\(\partial\mathcal L_{\mathrm{BCE}}/\partial v=-y/v+(1-y)/(1-v)\)\(g_v=1\times[-y/v+(1-y)/(1-v)]=-y/v+(1-y)/(1-v)\)
Sigmoid\(\partial v/\partial z=v(1-v)\)\(g_z=g_v\times v(1-v)=[-y/v+(1-y)/(1-v)]\,v(1-v)=-y(1-v)+(1-y)v=v-y\)
Add \(b\)\(\partial z/\partial s=1,\;\partial z/\partial b=1\)\(g_s=g_b=g_z\times1=(v-y)\times1=v-y\)
Multiply: to \(h\)\(\partial s/\partial h=w_3\)\(g_h=g_s\times w_3=(v-y)\times w_3=(v-y)w_3\)
Multiply: \(w_3\)\(\partial s/\partial w_3=h\)\(g_{w_3}=g_s\times h=(v-y)\times h=(v-y)\max(0,x_1w_1+x_2w_2),\quad w_3\leftarrow w_3-\eta g_{w_3}\)
Max / ReLU\(\partial h/\partial a=\mathbf1[a>0]\)\(g_a=g_h\times\mathbf1[a>0]=[(v-y)w_3]\times\mathbf1[a>0]=(v-y)w_3\mathbf1[x_1w_1+x_2w_2>0]\)
Add inputs\(\partial a/\partial u_1=\partial a/\partial u_2=1\)\(g_{u_1}=g_{u_2}=g_a\times1=[(v-y)w_3\mathbf1[a>0]]\times1=(v-y)w_3\mathbf1[x_1w_1+x_2w_2>0]\)
Multiply \(w_1\)\(\partial u_1/\partial w_1=x_1\)\(g_{w_1}=g_{u_1}\times x_1=[(v-y)w_3\mathbf1[a>0]]\times x_1=(v-y)w_3\mathbf1[x_1w_1+x_2w_2>0]x_1\)
Multiply \(w_2\)\(\partial u_2/\partial w_2=x_2\)\(g_{w_2}=g_{u_2}\times x_2=[(v-y)w_3\mathbf1[a>0]]\times x_2=(v-y)w_3\mathbf1[x_1w_1+x_2w_2>0]x_2\)
From scalars to vectors
Previous examples used scalars. Our network uses vectors and matrices.
| Mapping | Derivative is a… | Shape |
|---|---|---|
| Scalar → Scalar | A number | \(1\) |
| Vector → Scalar | Gradient | Same as input: \(n\times1\) |
| Vector → Vector | Jacobian matrix | \(m\times n\) |
\(n\) inputs, \(m\) outputs. Vectors are columns; \(J_{ij}=\partial f_i/\partial x_j\).
- Same chain rule, larger objects. Each node now propagates gradients through its local Jacobian instead of a scalar derivative.
- One scalar loss, gradients for every parameter. Backpropagation computes \(\nabla_\theta\mathcal L\); each parameter's gradient has the same shape as that parameter.
Deep Learning Frameworks
Frameworks compute gradients automatically using computational graphs.
Dynamic graph
Build as the code runs
Run operations→Record graph→Backward
- Pros: Flexible control flow; easy to debug.
- Cons: Graph-recording overhead on each run.
- Cons: Less scope for whole-graph optimization.
Static graph
Define or trace, then execute
Build / trace→Optimize→Run / reuse
- Pros: Optimize once; reuse the graph.
- Cons: Tracing or compilation adds startup cost.
- Cons: Harder to debug; new shapes may retrace.
Theano
StaticSymbolic expressions → compiled function
Caffe
StaticConfiguration-defined layer graph
TensorFlow 1.x
StaticDefine graph → run a Session
Chainer
DynamicDefine-by-run; record executed operations
PyTorch
DynamicEager autograd; rebuild each forward run
JAX
Static (traced)†Trace a function → compile and reuse
† JAX traces tensor operations into a graph for compilation and reuse with jax.jit; eager execution is also available.
Frameworks have evolved: modern TensorFlow and PyTorch support both dynamic graphs and static graph execution.
From \(y=wx+b\) to nn.Module
A linear layer computes \(wx+b\). For classification, call its output \(z\) (logit); \(y\) is the target.
1. Linear layer: \(z=wx+b\)
import torch
from torch import nn
class BinaryClassifier(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(1, 1)
def forward(self, x):
return self.linear(x) # logits
x = torch.tensor([[0.5], [1.0]])
y = torch.tensor([[1.0], [0.0]])nn.Linear(1, 1) learns one weight \(w\) and one bias \(b\). forward(x) returns \(z=wx+b\).
Set up training
model = BinaryClassifier()
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.SGD(
model.parameters(), lr=0.1)2. Forward pass
# Clear old gradients; backward accumulates.
optimizer.zero_grad()
logits = model(x)
loss = criterion(logits, y)model(x) calls forward(). BCEWithLogitsLoss includes sigmoid internally.
3. Backward pass
loss.backward()Computes parameter gradients in .grad. No custom backward() is needed for this model.
4. Update parameters
optimizer.step()Uses the gradients to update weights and bias. Backward alone does not update parameters.
Start with tensors: \(z=wx+b\)
One sample. Scalar tensors. No model class or optimizer yet.
Python codeComputational graph
1. Create tensors
import torch
x = torch.tensor(2.0)
y = torch.tensor(1.0)
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)Track gradients for \(w,b\), not the fixed input or target.
\(x=2\)\(w=0.5\)\(b=0\)\(y=1\)
A Tensor stores a value. requires_grad=True tells autograd to track operations involving it.
2. Multiply, then add
u = w * x
z = u + bPyTorch records these operations automatically.
\(w,x\)→\(\times\)→\(u,b\)→\(+\)→\(z\)
\(u=0.5\times2=1,\qquad z=1+0=1\).
3. Compute the loss
loss = torch.nn.BCEWithLogitsLoss()(z, y)This function combines Sigmoid and BCE.
\(z,y\)→Sigmoid + BCE→\(\mathcal L\)
\(v=\sigma(1)\approx0.7311,\qquad\mathcal L=-\log v\approx0.3133\).
4. Compute gradients
loss.backward()
print(w.grad, b.grad)Gradients are stored in .grad. \(w,b\) do not change.
\(g_w\)
\(\times\)
\(+\)
Sigmoid + BCE
\(\mathcal L\)
\(1\)
\(g_z=v-y\)
\(g_u=g_z\times1\)
\(g_w=g_u x\approx-0.5379\)
\(g_b=g_z\times1\approx-0.2689\)
Inside an Add or Multiply node
Add: \(z=x+y\)
import torch
class Add(torch.autograd.Function):
@staticmethod
def forward(ctx, x, y):
return x + y
@staticmethod
def backward(ctx, grad_z):
return grad_z, grad_z\(x\)
\(y\)
\(+\)
\(z=x+y\)
\(\mathcal L\)
\(g_z\times1\)
\(g_z\times1\)
\(g_z\)
Saved for backward: nothing. Both local derivatives are \(1\).
return grad_z, grad_z → gradients for \(x\), then \(y\).
Multiply: \(z=xy\)
class Multiply(torch.autograd.Function):
@staticmethod
def forward(ctx, x, y):
ctx.save_for_backward(x, y)
return x * y
@staticmethod
def backward(ctx, grad_z):
x, y = ctx.saved_tensors
return grad_z * y, grad_z * x\(x\)
\(y\)
\(\times\)
\(z=xy\)
\(\mathcal L\)
\(g_z\times y\)
\(g_z\times x\)
\(g_z\)
Forward saves \(x,y\). Backward retrieves them to use local derivatives \(y,x\).
save_for_backward(x, y) → ctx.saved_tensors
requires_grad=True: request gradients
- Track operations: set
requires_grad=Trueon the tensors you want gradients for (with grad mode enabled). - Read gradients:
loss.backward()accumulates gradients inx.gradfor a tracked leaf tensorxused in the loss. - Use the custom node: call
Add.apply(x, y).grad_zis the incoming gradient; return gradients forx, theny.
From operations to models
Operations in PyTorch
add · sub · mul · div · powArithmeticmatmul · mm · bmm · einsumMatrix and tensor productsexp · log · sqrt · sin · cosMathematical functionssum · mean · max · minReductionsrelu · gelu · sigmoid · softmaxActivations and probabilitieslayer_norm · batch_norm · conv2dNormalization and convolutionreshape · transpose · cat · stackTensor rearrangement… and many more.
Autograd provides local backward rules for these operations. Differentiable tensor values must have gradient tracking enabled.
# 1. Tensor operations: matrix multiply + bias
import torch
from torch import nn
def affine(x, weight, bias):
return x @ weight.T + bias
# 2. Package operations and trainable parameters
class Linear(nn.Module):
def __init__(self, n_in, n_out):
super().__init__()
self.weight = nn.Parameter(
torch.randn(n_out, n_in) * n_in ** -0.5)
self.bias = nn.Parameter(torch.zeros(n_out))
def forward(self, x):
return affine(x, self.weight, self.bias)
# 3. Linear projections + attention operations
class Attention(nn.Module):
def __init__(self, d, heads):
super().__init__()
self.heads, self.dk = heads, d // heads
self.qkv, self.out = Linear(d, 3 * d), Linear(d, d)
def forward(self, x):
B, T, D = x.shape
qkv = self.qkv(x).reshape(B, T, 3, self.heads, self.dk)
q, k, v = [t.transpose(1, 2) for t in qkv.unbind(2)]
scores = (q @ k.transpose(-2, -1)) / self.dk ** 0.5
weights = scores.softmax(dim=-1)
h = (weights @ v).transpose(1, 2).reshape(B, T, D)
return self.out(h)
# 4. Attention + MLP + residuals + normalization
class Block(nn.Module):
def __init__(self, d, heads):
super().__init__()
self.attn = Attention(d, heads)
self.norm1, self.norm2 = nn.LayerNorm(d), nn.LayerNorm(d)
self.mlp = nn.Sequential(
Linear(d, 4 * d), nn.GELU(), Linear(4 * d, d))
def forward(self, x):
x = x + self.attn(self.norm1(x))
return x + self.mlp(self.norm2(x))
# 5. Embeddings + repeated blocks + output head
class Model(nn.Module):
def __init__(self, vocab=1000, d=64, heads=4, depth=6):
super().__init__()
self.token = nn.Embedding(vocab, d)
self.position = nn.Embedding(128, d)
self.blocks = nn.ModuleList(
[Block(d, heads) for _ in range(depth)])
self.head = Linear(d, vocab)
def forward(self, ids):
positions = torch.arange(ids.size(1), device=ids.device)
x = self.token(ids) + self.position(positions)
for block in self.blocks:
x = block(x)
return self.head(x)
model = Model() # bidirectional attention; sequences <= 128
What does this mean for adversarial examples?
Same model, same chain rule. Different variable: differentiate with respect to the perturbation.
1. Add a random perturbation
\[x_{\mathrm{adv}}=x+\delta\]
Start with \(\delta_0\sim\mathcal U[-\epsilon,\epsilon]\) elementwise. Keep \(x,y,\theta\) fixed.
2. Compute the loss
\[\mathcal L(\delta)=\ell\!\left(f_\theta(x+\delta),y\right)\]
The example uses a continuous input and a classification loss.
3. Differentiate with respect to \(\delta\)
\[\nabla_\delta\mathcal L=I^\top\nabla_{x_{\mathrm{adv}}}\mathcal L=\nabla_{x_{\mathrm{adv}}}\mathcal L\]
Evaluate at the random start \(\delta_0\), not at zero.
4. Sign step, then project
\[\begin{aligned}\delta_1&=\operatorname{clip}\!\left(\delta_0+\epsilon\,\operatorname{sign}(\nabla_\delta\mathcal L),-\epsilon,\epsilon\right)\\x_{\mathrm{adv}}&=x+\delta_1,\qquad\|\delta_1\|_\infty\le\epsilon\end{aligned}\]
PyTorch Implementation
import torch
from torch import nn
# Assume model, x, y, and eps are prepared
model.eval()
model.requires_grad_(False) # freeze parameters
x = x.detach() # keep the original input fixed
delta = torch.empty_like(x).uniform_(-eps, eps)
delta.requires_grad_()
# Keep the path from delta to the loss
x_adv = x + delta
logits = model(x_adv)
loss = nn.CrossEntropyLoss()(logits, y)
loss.backward()
grad_delta = delta.grad # gradient stored on the leaf
# One projected random-start sign step
with torch.no_grad():
delta = delta + eps * grad_delta.sign()
delta = delta.clamp(-eps, eps)
x_adv = x + delta
Random-start FGSM. One random start and one projected sign step. Standard FGSM starts from zero. Apply valid-input bounds when required.
PGD: multi-step FGSM
1. Initialize the perturbation
\[\delta^{(0)}\sim\mathcal U[-\epsilon,\epsilon],\quad x_{\mathrm{adv}}^{(0)}=x+\delta^{(0)}\]
Keep x and the model fixed. Project the initial perturbation to valid inputs.
2. Differentiate with respect to delta
\[g^{(k)}=\nabla_\delta\ell(f_\theta(x+\delta^{(k)}),y)\]
The code requests the gradient for delta, not the model parameters.
3. Update delta, then project
\[\begin{aligned}\delta^{(k+1)}&=\Pi_{\mathcal D}\!\left(\delta^{(k)}+\alpha\operatorname{sign}(g^{(k)})\right)\\\mathcal D&=\{\delta:\|\delta\|_\infty\le\epsilon,\;0\le x+\delta\le1\}\\x_{\mathrm{adv}}&=x+\delta^{(T)}\end{aligned}\]
Only delta changes. x_adv is computed from x plus delta.
PyTorch Implementation
import torch
from torch.nn import functional as F
# model, x, y, eps, alpha, steps are prepared
model.eval().requires_grad_(False)
x0 = x.detach() # fixed original input
delta = torch.empty_like(x0).uniform_(-eps, eps)
delta = (x0 + delta).clamp(0, 1) - x0
for _ in range(steps):
delta = delta.detach().requires_grad_(True)
x_adv = x0 + delta
loss = F.cross_entropy(model(x_adv), y)
loss.backward()
grad = delta.grad
with torch.no_grad():
delta = delta + alpha * grad.sign()
delta = delta.clamp(-eps, eps)
delta = (x0 + delta).clamp(0, 1) - x0
x_adv = (x0 + delta).detach()
PGD = gradient → sign step → projection → repeat.
MI-FGSM: accumulate a direction
1. Initialize
\[\delta^{(0)}=0,\quad g^{(0)}=0,\quad\alpha=\epsilon/T\]
Start with no perturbation and no momentum.
2. Accumulate delta gradients
\[\begin{aligned}r^{(k)}&=\nabla_\delta\ell(f_\theta(x+\delta^{(k)}),y)\\g^{(k+1)}&=\mu g^{(k)}+\frac{r^{(k)}}{\|r^{(k)}\|_1}\end{aligned}\]
momentum stores g. A zero gradient adds zero.
3. Update delta using momentum
\[\begin{aligned}\delta^{(k+1)}&=\Pi_{\mathcal D}\!\left(\delta^{(k)}+\alpha\operatorname{sign}(g^{(k+1)})\right)\\x_{\mathrm{adv}}&=x+\delta^{(T)}\end{aligned}\]
Use the PGD constraint set D. Keep the original x unchanged.
PyTorch Implementation
import torch
from torch.nn import functional as F
# One sample; model, x, y, eps, mu, steps exist
model.eval().requires_grad_(False)
x0 = x.detach()
delta = torch.zeros_like(x0)
momentum = torch.zeros_like(x0)
alpha = eps / steps
for _ in range(steps):
delta = delta.detach().requires_grad_(True)
loss = F.cross_entropy(model(x0 + delta), y)
grad, = torch.autograd.grad(loss, delta)
norm = grad.abs().sum()
momentum = mu * momentum + grad / norm.clamp_min(1e-12)
with torch.no_grad():
delta = (delta + alpha * momentum.sign()).clamp(-eps, eps)
delta = (x0 + delta).clamp(0, 1) - x0
x_adv = (x0 + delta).detach()
MI-FGSM follows accumulated gradients.
Carlini–Wagner with SGD
1. Compare logits at x + delta
\[m_y(x+\delta)=Z_y(x+\delta)-\max_{j\ne y}Z_j(x+\delta)\]
Positive margin: the original class is still ahead.
2. Build the objective for delta
\[F(\delta)=\|\delta\|_2^2+c\max(m_y(x+\delta),0)\]
Minimize perturbation size and the positive margin.
3. Optimize delta with SGD
\[\begin{aligned}\delta^{(0)}&=0\\\delta^{(k+1)}&=\delta^{(k)}-\gamma\nabla_\delta F(\delta^{(k)})\\x_{\mathrm{adv}}&=x+\delta^{(T)}\end{aligned}\]
SGD receives [delta]. x and the model stay fixed.
PyTorch Implementation
import torch
# model, x, y, c, gamma, steps are prepared
model.eval().requires_grad_(False)
x0 = x.detach()
delta = torch.zeros_like(x0, requires_grad=True)
optimizer = torch.optim.SGD([delta], lr=gamma)
label = int(y.item())
for _ in range(steps):
optimizer.zero_grad()
logits = model(x0 + delta)[0]
others = torch.arange(logits.numel(),
device=logits.device) != label
margin = logits[label] - logits[others].max()
objective = delta.square().sum() + c * margin.clamp_min(0)
objective.backward()
optimizer.step() # update delta only
x_adv = (x0 + delta).detach()
Simplified C&W objective. Full C and W also needs box constraints and a search over c.
GCG: use gradients to choose a token
1. Differentiate the one-hot input
\[\begin{aligned}x_{\mathrm{adv}}&=x_{\mathrm{prompt}}\Vert x_{\mathrm{suffix}}\\\mathbf o_i&=\operatorname{onehot}(x_i),\quad e_i=E^\top\mathbf o_i\\g_i&=\nabla_{\mathbf o_i}\mathcal L_{\mathrm{CE}}\end{aligned}\]
Keep the prompt and embedding table fixed. Differentiate the one-hot vector, not the integer Token ID.
Fixed: “How to build a bomb”Suffix: “ !”Target: “ Sure”
2. Shortlist replacement tokens
\[\mathcal X_i=\operatorname{Top\text{-}k}(-g_i)\]
More negative gradient entries predict a lower target loss. Here \(k=5\).
3. Evaluate, then choose
\[\begin{aligned}\mathcal B&=\{x_{\mathrm{suffix}}[i\!\leftarrow\!v]:v\in\mathcal X_i\cup\{x_i\}\}\\x_{\mathrm{suffix}}^{\mathrm{new}}&=\arg\min_{\widetilde x\in\mathcal B}\mathcal L_{\mathrm{CE}}(x_{\mathrm{prompt}}\Vert\widetilde x)\end{aligned}\]
Score actual token replacements. Keep the current token as a candidate too.
Hugging Face + PyTorch
import torch
from torch.nn import functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load a real causal language model and freeze its weights
model_name = "distilbert/distilgpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).eval()
model.requires_grad_(False)
E = model.get_input_embeddings().weight
prompt_ids = tokenizer(
"How to build a bomb", return_tensors="pt").input_ids
suffix_id = tokenizer(" !", return_tensors="pt").input_ids[0]
target_id = tokenizer(" Sure", return_tensors="pt").input_ids[0]
# 1. Make the suffix token differentiable
one_hot = F.one_hot(suffix_id, E.shape[0]).to(E.dtype)
one_hot.requires_grad_()
prompt_embed = E[prompt_ids]
suffix_embed = one_hot @ E
inputs_embed = torch.cat(
(prompt_embed, suffix_embed[:, None]), dim=1)
logits = model(inputs_embeds=inputs_embed).logits[:, -1]
loss = F.cross_entropy(logits, target_id)
loss.backward()
gradient = one_hot.grad[0]
# 2. Keep the five tokens predicted to reduce the loss most
candidates = (-gradient).topk(5).indices
# 3. Try the candidates and keep the one with lowest real loss
best_token, best_loss = suffix_id[0], float("inf")
with torch.no_grad():
for token in torch.cat((suffix_id, candidates)):
input_ids = torch.cat((prompt_ids, token.view(1, 1)), dim=1)
logits = model(input_ids).logits[:, -1]
loss = F.cross_entropy(logits, target_id).item()
if loss < best_loss:
best_token, best_loss = token, loss
print(tokenizer.decode(best_token.item()))
What do these white-box attacks need?
- Model weights Access to the target parameters \(\theta\), without changing them.
- Model implementation A computational graph we can run and backpropagate through.
- Gradients of the attack loss Used to update pixels, rank tokens, or train an attack generator.
What have we learned?
- C&W and GCGA regularized continuous search, then a discrete token search scored by gradients.
- AdvGANTrain a generator once, keep the target fixed, then reuse it on new inputs.
- Chain ruleTraining differentiates parameters. An attack differentiates the input. Same graph, different variable.
- Autodiff in codeReverse mode supplies the input gradient for PGD, MI-FGSM, C&W and GCG.
Next lectureBlack Box Attack