DDPG on Reacher: Twenty Arms Chasing a Balloon

Reacher was the easiest of the three environments in this series. It also taught me the most, because what worked here was the exact opposite of what worked on Tennis a year earlier.

The environment has twenty robot arms. Each arm has two joints and its own balloon that floats around it. An arm earns a reward for every step its hand stays inside the balloon, so the job is to follow the balloon as it moves.

The code is in the repo.

The environment¶

Each arm gets 33 numbers: where its parts are, how they are rotated and how fast they are moving. It controls 4 values, which are the turning forces on its two joints. Those are smooth values, not a menu of buttons, so DDPG is the right tool and DQN is not.

The task is solved when the average score across all twenty arms is +30 or more over 100 episodes in a row.

The twenty arms are the interesting part. They don’t compete and they don’t cooperate. They are twenty copies of the same problem running side by side, and they all pour their experience into one shared replay buffer (the memory the agent learns from). That helps in two ways:

• The agent collects experience twenty times faster.
• Each batch it learns from is more varied, because it comes from twenty arms in twenty different positions.

That shared memory is a big part of why this went so smoothly.

The model¶

The model has the same shape as the Tennis agents. The actor looks at the state and says what to do. The critic rates that move. Each network has six fully-connected layers:

• Actor: 400 → 300 → 200 → 100 → 50 units with ReLU, and a tanh output. The tanh keeps the joint forces inside their allowed range.
• Critic: The state goes through 400 units. The action joins at the second layer. Then the layers get smaller and end in a single number.

The settings are:

• Replay buffer: 1e6
• Batch size: 1024
• Gamma: 0.99
• Tau: 1e-3
• Actor LR: 1e-4, critic LR: 1e-3
• Weight decay: 0, optimizer AdamW
• Exploration noise: Ornstein-Uhlenbeck

The exploration noise¶

To learn, the agent has to try moves it wouldn’t normally pick. The usual trick is to add a little random noise to its actions.

My first version used smaller, three-layer networks and plain Gaussian noise, which is a fresh random nudge at every step. After 2,000 episodes it still hadn’t reached the target. Deeper networks helped a little, but 1,000 episodes later the task was still unsolved.

Then I switched to Ornstein-Uhlenbeck (OU) noise, and the task was solved in 101 episodes.

On Tennis, the same switch did the opposite. OU noise stalled training there, and Gaussian noise fixed it. Here is why I think that happens. OU noise is “sticky”: each nudge is similar to the one before, so it keeps pushing the agent in the same direction for a while. An arm reaching for a target wants exactly that, because a good reach is one smooth motion. A player reacting to a tennis ball does not.

I don’t have a general rule to offer, other than to try both. And remember that the default settings in a paper were tuned for that paper’s task. They may not suit yours.

Results¶

cd scripts
python train_agent.py --unity-app Reacher.app --target-score 30

Episode 95  Average Score: 29.26
Episode 96  Average Score: 29.35
Episode 97  Average Score: 29.43
Episode 98  Average Score: 29.52
Episode 99  Average Score: 29.61
Episode 100 Average Score: 29.69
Episode 101 Average Score: 30.08

Environment solved in 101 episodes! Average Score: 30.08

The agent solved the task in 101 episodes. This curve looks nothing like the other two. It shoots up in the first thirty episodes and then stays near the top. By that point the agent is already good. The 100-episode average just needs time to catch up.

Reacher pays out a reward on almost every step, so there is no long stretch of zeros like the one in Tennis. The agent always has something to learn from.

This video shows the untrained arms and then the trained ones. The untrained arms flail around. The trained arms find their balloons and stay on them:

Links¶

• Repo: github.com/sina5/reacher
• Report — full architecture and hyperparameters
• Continuous Control with Deep Reinforcement Learning

The next method I want to try is D4PG. The ddpg and mlagents scripts come from the Udacity deep-reinforcement-learning repo (MIT).