Sina's Blog

CS and AI

2 min readAIProgramming

Training a DQN Agent to Collect Bananas in Unity

This was the first RL agent I trained end to end. The environment is Unity’s Banana Collector: a square arena full of yellow and blue bananas. Yellow is worth +1, blue is worth −1, and the agent has to work out which is which from a stream of numbers.

Code is in the repo. Here’s how it went.

The environment

The agent gets 37 numbers per step — its own velocity plus a ray-cast view of what’s in front of it — and picks one of four moves: forward, backward, turn left, turn right. That’s it.

To count as solved, it has to average +13 over 100 episodes in a row. Not one good run, a hundred.

The model

Since the state is already a feature vector, there’s nothing to convolve over. A small fully-connected Q-network does the job — 37 in, 4 action values out, with two hidden layers of 64 units and ReLU in between. That’s a tiny network, and it was enough.

The settings I ended up with:

  • Replay buffer: 1e5
  • Minibatch: 64
  • Gamma: 0.99
  • Tau: 1e-3
  • Learning rate: 5e-4
  • Update the network every 4 steps

Two things are carrying that setup. The replay buffer keeps old transitions around and samples them randomly, so the network isn’t learning from a hundred nearly-identical consecutive frames. And the target network — a slow copy of the Q-network, dragged along at tau = 1e-3 — gives you a target that isn’t moving every time you take a gradient step. Take either one out and training falls apart.

Training

cd scripts
python train_agent.py --unity-app Banana.app --target-score 13

Episode 100     Average Score: 1.065
Episode 200     Average Score: 4.18
Episode 300     Average Score: 7.93
Episode 400     Average Score: 11.20
Episode 481     Average Score: 13.07
Environment solved in 481 episodes!     Average Score: 13.07

Average score per episode over training, climbing steadily past +13 at episode 481.

481 episodes. The thing I didn’t expect was how boring the curve is — it just goes up. No plateau, no sudden jump where it figures something out. It’s grinding out a better value estimate the whole way.

To watch a trained agent next to an untrained one:

python test_agent.py --unity-app Banana.app --checkpoint-file ../checkpoints/checkpoint_481.pth

What I’d add

Three things I left out and would do next: Double DQN, which stops the max operator from systematically overestimating values; Dueling DQN, which splits the network into a state-value stream and an advantage stream; and prioritized replay, so surprising transitions get sampled more than boring ones.

Setup instructions and the Unity downloads for each platform are in the repo:

The dqn and mlagents scripts started from Udacity’s deep-reinforcement-learning repo (MIT) with small changes.

Related posts