Small Vision Transformers: A Tiny Diffusion Model on CIFAR-10
A 9.8 M parameter Diffusion Transformer learns to draw CIFAR-10 images in 86 minutes on one GPU. The loss is flat long before the samples look right.
Lead AI Engineer · Writing since 2016
I write about reinforcement learning, transformers, multi-agent systems, and the tools I build along the way.
A 9.8 M parameter Diffusion Transformer learns to draw CIFAR-10 images in 86 minutes on one GPU. The loss is flat long before the samples look right.
ViT-B/16 on Flowers-102 with 10 images per class: 39.0% from scratch, 98.4% with a linear probe and 99.2% with LoRA, plus a check for leaked test images.
A small ViT trained on CIFAR-10 learns local attention in layer 1 and global attention in the last layers. The fixes that raise accuracy make it more local.
A small Vision Transformer starts at 57% on CIFAR-10 and reaches 92.9% after eight fixes. Each fix has a measured gain, and ResNet-18 still wins.
Sabattier is a free batch photo editor for Windows and macOS with auto enhance, masks, portrait retouching and background removal, all on your computer.
Mullion FileMan is a C++17 dual-pane terminal file manager. It has twenty folder_* CLI tools that compare and sync folders, and an MCP server for AI agents.
Redthread keeps your AI coding agent's memory in a git repo and serves it over MCP, so Claude Code and other agents remember on every machine.
Read in order
Multi-part guides that build on each other, start to finish.
Subscribe to the RSS feed, or follow along on GitHub, LinkedIn and the RL Factory YouTube channel.