Training a DQN Agent to Collect Bananas in Unity

This is the first reinforcement learning (RL) agent that I trained from start to end. The environment is the Unity Banana Collector. It is a square area with yellow bananas and blue bananas. A yellow banana gives a reward of +1. A blue banana gives a reward of −1. The agent gets only a list of numbers, and it must learn which banana is which.

The code is in the repo.

The environment¶

At each step, the agent gets 37 numbers. These numbers give the velocity of the agent and a ray-cast view of the area in front of the agent. The agent selects one of four actions:

• Move forward.
• Move backward.
• Turn left.
• Turn right.

The task is complete when the average score is +13 or more for 100 episodes in sequence. One good episode is not sufficient.

The model¶

The state is already a feature vector, thus a convolutional network is not necessary. A small fully-connected Q-network is sufficient. It has 37 inputs and 4 outputs, one value for each action. It has two hidden layers with 64 units each, and a ReLU function between the layers.

The settings are:

• Replay buffer: 1e5
• Minibatch: 64
• Gamma: 0.99
• Tau: 1e-3
• Learning rate: 5e-4
• Network update: every 4 steps

Two parts of this setup are important:

• The replay buffer. It keeps old transitions and gives random samples from them. Thus, the network does not learn from many frames in sequence that are almost the same.
• The target network. It is a slow copy of the Q-network, with an update rate of tau = 1e-3. It gives a target that does not change at each gradient step.

If you remove one of these two parts, the training does not converge.

Training¶

cd scripts
python train_agent.py --unity-app Banana.app --target-score 13

Episode 100     Average Score: 1.065
Episode 200     Average Score: 4.18
Episode 300     Average Score: 7.93
Episode 400     Average Score: 11.20
Episode 481     Average Score: 13.07
Environment solved in 481 episodes!     Average Score: 13.07

The agent completed the task in 481 episodes. The curve increases steadily from start to end. It does not stop at a level, and it does not jump suddenly. The value estimate of the agent becomes better at each step.

To compare a trained agent with an untrained agent, use this command:

python test_agent.py --unity-app Banana.app --checkpoint-file ../checkpoints/checkpoint_481.pth

Possible improvements¶

This project does not use the three methods that follow. They are the next steps:

• Double DQN. It prevents the max operator from giving values that are too high.
• Dueling DQN. It divides the network into two streams: one for the state value and one for the advantage.
• Prioritized replay. It samples unexpected transitions more frequently than usual transitions.

The repo has the setup instructions and the Unity downloads for each platform:

• Repo: github.com/sina5/banana_collector
• Report

The dqn and mlagents scripts come from the Udacity deep-reinforcement-learning repo (MIT), with small changes.