Small Vision Transformers: Reward Fine-Tuning a Diffusion Model with DDRL

RL fine-tuning of a DiT with a ViT reward and a held-out judge. Data regularization wins on reward and FID; a 10x learning rate shows reward hacking.

On this page

So far in this series, the diffusion models learned only from data. This post adds a reward. I take the DDPM model from an earlier post and fine-tune it with reinforcement learning, so its images look more like their class to a classifier.

The risk with any reward is reward hacking: the model finds images that score well with the reward model but aren’t better. So I check every run with a second classifier that the fine-tuning never sees.

The reward and the judge

  • Reward: log p(class | image) from a frozen ViT. It’s the “patch 4, 2x epochs” model from the first post, with 93.1% test accuracy on CIFAR-10.
  • Judge: a ResNet-18 that I trained for 30 epochs on CIFAR-10, with 92.6% test accuracy. The fine-tuning never uses it.

If the reward goes up and the judge’s accuracy goes up too, the model probably got better at drawing its class. If the reward goes up and the judge goes down, the model is fooling the reward ViT.

I also track FID-10k and the ordinary diffusion loss on held-out test images. A rise in that loss means the model has moved away from the data.

How the fine-tuning works

Each iteration samples images and rewards the good ones:

  1. Pick 16 class labels. Make 8 images for each label (128 images).
  2. Sample each image with stochastic DDIM: 20 steps, fresh noise at each step, and no guidance. Each step is then a Gaussian with a known density.
  3. Score each image with the reward ViT.
  4. Normalize the rewards inside each group of 8. The result is the advantage: how much better an image is than the others with the same label.
  5. Raise the log-probability of each sampling step in proportion to its advantage, on every second step.

The loss is −advantage × log p(x_s | x_t). This follows Algorithm 1 of DDRL (Ye et al., 2025). Every run has 300 iterations, AdamW, gradient clipping at norm 1, and starts from the EMA weights of the DDPM model.

What stops the model from drifting too far is the regularizer. I compare three:

  • Reward only: no regularizer. This is the reward-hacking baseline.
  • Reverse KL: a penalty on the distance between each sampling step of the model and the same step of the frozen base model, with weight beta = 0.1. Many RL methods for diffusion models use a KL penalty of this kind.
  • Data (DDRL): the ordinary diffusion loss on real training images, with weight beta. The paper shows that this acts as a KL to the data distribution, so it needs no frozen copy of the base model. I use one random timestep per real image.

The base model

Without guidance and with only 20 stochastic steps, the base model is weaker than in the earlier posts. On 10,000 samples, it has FID-10k 51.9, reward-ViT accuracy 60.1%, judge accuracy 50.8%, and a mean reward of −1.195.

Results at learning rate 1e-5

Run Reward Reward-ViT accuracy Judge accuracy FID-10k Test diffusion loss
Base model −1.195 60.1% 50.8% 51.9 0.0323
Reward only −1.034 66.3% 58.8% 47.3 0.0325
Reward only, seed 1 −1.062 65.0% 56.4% 43.7 0.0327
Reverse KL, beta 0.1 −1.084 64.3% 55.2% 49.6 0.0323
Data, beta 0.01 −0.991 67.6% 60.2% 38.6 0.0324
Data, beta 0.01, seed 1 −0.927 70.2% 63.2% 38.2 0.0325
Data, beta 0.1 −0.986 67.8% 59.5% 41.6 0.0323
Data, beta 1 −1.051 65.4% 55.8% 46.1 0.0322

All rows are the final evaluation on 10,000 samples after 300 iterations.

At this learning rate, I see no reward hacking. In every run, the judge’s accuracy rises along with the reward, and FID goes down, not up. My guess is that the base model has no guidance, and the reward pushes each image toward its class, which is close to what guidance does. That would improve FID too.

Data regularization with a small beta (0.01 or 0.1) beats reward only and reverse KL on reward, on judge accuracy and on FID. Beta 0.01 has the lowest FID, and it’s also the most stable across the two seeds: FID 38.6 and 38.2. Reward only gives 47.3 and 43.7. Beta 0.01 and 0.1 end close in reward; beta 1 pulls harder toward the data loss, slows the reward gain and ends close to reward only.

Reverse KL keeps the model closest to the base: its test loss is the same as the base model’s, and it has the smallest gains on every measure.

Final FID-10k against final reward for each run. The base model is at the left with FID 51.9. The data-regularized runs with beta 0.01 are at the lower right, with the highest reward and FID near 38. The run with data beta 0.01 and learning rate 1e-4 is high above the others, with FID 92.4. A note says that the reward-only run at learning rate 1e-4 is off the scale, with reward −2.40 and FID 379.4.

Results at learning rate 1e-4

To push the methods harder, I ran reward only and data beta 0.01 again with a learning rate 10 times higher.

Run Reward Reward-ViT accuracy Judge accuracy FID-10k Test diffusion loss
Base model −1.195 60.1% 50.8% 51.9 0.0323
Reward only, lr 1e-4 −2.400 10.5% 10.5% 379.4 0.2380
Data, beta 0.01, lr 1e-4 −1.062 65.4% 45.7% 92.4 0.0435

Reward only collapsed. Both classifiers are at 10.5%, which is chance for 10 classes, and the images are a brown dot pattern. That’s not reward hacking, since the reward fell too. It’s training that broke.

The data-regularized run is the interesting one. The reward ViT says it got better: accuracy 65.4%, up from 60.1%. The judge says it got worse: 45.7%, down from 50.8%. FID rose from 51.9 to 92.4, and the test loss rose by a third. That’s reward hacking, and only the held-out checks caught it.

Four charts of reward and judge accuracy against iteration, evaluated every 25 iterations. Top row, learning rate 1e-5: all runs rise slowly in both reward and judge accuracy. Bottom row, learning rate 1e-4: reward only falls to −2.4 and 10% judge accuracy after iteration 200. Data beta 0.01 keeps a reward near −1.0, but its judge accuracy falls from 57% at iteration 175 to 46% at iteration 300.

The curves show when it happened. For data beta 0.01 at lr 1e-4, judge accuracy peaked at 57.3% at iteration 175 and then fell to 46.0% by iteration 300. From iteration 150 on, the reward stayed near −1.0. If I’d watched only the reward, I’d have kept that run.

Samples from the same noise in five panels: the base model, reward only and data beta 0.01 at learning rate 1e-5, then data beta 0.01 and reward only at learning rate 1e-4, all after 300 iterations. The first three panels look almost the same. The fourth has blocky shapes, strong colors and a fine dot pattern. The fifth is brown dot texture with no objects.

At lr 1e-5, the samples barely change by eye after 300 iterations, even though the measures move. At lr 1e-4 with data regularization, the shapes become blocky and a fine dot pattern appears. That pattern seems to be what the reward ViT likes and the judge doesn’t.

Cost

Each run took 39 to 51 minutes on one RTX 3080 Ti, without the evaluations: 7.8 seconds per iteration for reward only, 8.1 to 8.6 seconds with data regularization, and 10.1 seconds with reverse KL, which runs the frozen base model at every step.

Limits

  • I ran reward only and data beta 0.01 at lr 1e-5 with two seeds. I ran every other setting once, including both lr 1e-4 runs, so those results can change with another seed.
  • Each run had only 300 iterations. With more iterations, the runs at lr 1e-5 could also start to hack the reward.
  • The curves use FID on 2,000 samples, which is biased upward. For the same base model, FID-2k is 63.1 and FID-10k is 51.9. Use the curves for trends, and the tables for values.
  • The final FID uses 10,000 samples (FID-10k). FID from fewer samples is biased upward, so these values are higher than the FID-50k (50,000 samples) that papers report. Compare them with each other, not with published results.
  • The judge is a classifier too. Agreement between two classifiers isn’t the same as a person looking at the images.

Run it

  1. Train the judge.
  2. Evaluate the base model.
  3. Run one fine-tuning setting. Each setting writes to its own folder.
uv run python scripts/train_judge.py
uv run python scripts/finetune_reward.py --base-only
uv run python scripts/finetune_reward.py --regularizer data --beta 0.01

The code is in the repo.

What I take from this

  • With a small learning rate, reward fine-tuning improved this model by every measure I had, and the data regularizer improved it most.
  • The data regularizer needs no frozen copy of the model, and it was the most stable across seeds.
  • At a learning rate 10 times higher, the same method hacked the reward. A held-out judge, FID and the test loss caught it; the reward didn’t.
Written by Sina Fathi-Kazerooni

I write about reinforcement learning, transformers, multi-agent systems, and the tools I build along the way.